- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces ExamOCRBench, a benchmark for evaluating VLMs on real-world examination understanding and automated grading. The benchmark consists of 2,464 photographed primary-school examination pages across Chinese, Mathematics, and English, with human-verified annotations for document structure, handwritten responses, and grading decisions.
The benchmark includes four tasks: (1) Exam Paper Grading, assessing page-level grading accuracy and the ability to abstain when information is insufficient; (2) Structure Parsing, evaluating hierarchical document understanding; (3) Content Grounding, locating question and response regions; and (4) Handwritten Text Recognition, transcribing students' handwritten answers. The authors evaluate five VLMs and report that even the best-performing model achieves only 58.18% page-level grading accuracy, highlighting the challenges of automated grading on real-world examination documents.
Critical strengths
The benchmark addresses the practical problem of grading real-world handwritten examinations, incorporating realistic scenarios such as handwritten corrections, diverse document layouts, and incomplete responses. In particular, the inclusion of SafetyUnitRecall can be valuable for assessing whether models can recognize situations where grading is not possible due to insufficient information. Additionally, the benchmark covers four related tasks, allowing the evaluation of multiple capabilities relevant to examination understanding beyond grading accuracy alone.
Critical weaknesses
The related work overlooks two highly relevant studies: FERMAT[1] and VEHME[2]. FERMAT introduces a benchmark for evaluating error detection, localization, and correction in handwritten mathematical solutions, while VEHME develops an automated handwritten mathematics assessment framework and evaluates models using complementary metrics and controlled experimental analyses.
Although ExamOCRBench differs from these studies by focusing on full-page examination grading across multiple subjects, the paper does not sufficiently establish how its proposed evaluation framework advances beyond existing approaches.
In particular, the evaluation relies heavily on page-level score agreement, without sufficiently reporting response-level grading metrics or investigating the contribution of individual perception and reasoning capabilities to grading errors. Moreover, while structure parsing, grounding, and handwriting recognition are evaluated separately, their relationship to end-to-end grading performance remains largely unexplored.
Prior studies such as VEHME have considered error detection and localization, class imbalance, and controlled analyses of visual input conditions. The authors should discuss these methodological choices and justify why their proposed evaluation protocols adequately capture the capabilities necessary for reliable handwritten exam grading.
Overall, the current contribution appears closer to an extension of the evaluation domain and an aggregation of established tasks than to a substantially new evaluation methodology.
[1] Nath, Oikantik, et al. "Can Vision-Language Models Evaluate Handwritten Math?." Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025. [2] Nguyen, Thu Phuong, et al. "VEHME: A vision-language model for evaluating handwritten mathematics expressions." Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.
Questions for authors
-
How does ExamOCRBench differ substantially from prior work such as FERMAT and VEHME, beyond expanding the dataset and evaluation tasks?
-
How do the independently evaluated diagnostic tasks contribute to understanding end-to-end grading failures?
-
How does PageScoreAccuracy account for variations in the number of questions per page and the distribution of correct and incorrect responses? Could the authors provide response-level metrics such as accuracy and F1?
-
Could the authors provide the complete prompts, annotation guidelines, and inference configurations used in benchmark construction and evaluation to improve reproducibility?
-
Could the authors provide more complete benchmark samples and representative examples of model-generated solutions and grading decisions, including failure cases, to help readers better understand the evaluation process?
Thank you for the detailed report. We have updated our paper. We understand the judgement that the paper reads as a domain extension rather than a new methodology, and we do not try to argue it away; the revision tries to earn the methodological claim instead. Two facts frame the rest: the evaluation now covers nine VLMs and fifteen OCR models, and grading is strictly image-only (Sec. 4.1, App. B.1); the submitted Task 1 runs supplied auxiliary structure and text, and with App. F.1's stricter validity rule Astra is now at 47.22% page accuracy and 44.32% safety recall rather than 58.18 and 41.04.
W1 FERMAT and VEHME. Both are now in Sec. 2.3. FERMAT evaluates detection, localisation and correction of errors in handwritten mathematical solutions, including perturbed solutions for grades 7-12; VEHME studies handwritten mathematical expression assessment with training and visual prompting. Our setting differs on three axes that matter: the input is a photographed full page rather than a segmented solution, so the model must first decide what the response units are; the target is binary credit per response unit with an explicit abstention class, not error localisation inside one expression; and the protocol is a frozen benchmark with no training on the target distribution.
W2 Increment. We accept that the contribution is a benchmark and protocol, not a new model, and the revision says so plainly. Four elements are absent from FERMAT/VEHME. (i) Linked question/response identities let the same unit be scored on grading, parsing, grounding and recognition, so a failure is traced across tasks on one page (Figs. 11-13). (ii) 2,473 annotated insufficient-information units with recall and, new here, abstention precision, so "abstains when it should" is separated from "abstains indiscriminately". (iii) Conditions B-E intervene on structure, locations and transcriptions under one contract: transcriptions raise page accuracy by 3.49-12.71 points for all nine models, locations by 0.54-4.15, bounding the perceptual share. (iv) A directional decomposition of grading error: false credit is 15.6-48.4% against false rejection of 3.2-12.5%, so every model over-credits 3.4-10.2 times more often, and a further 4.2-13.3 points of apparently correct scores fail the reference-answer checks. This turns "the model is wrong" into how it is wrong, which is the kind of finding we believe a benchmark should deliver.
W3 Page-level agreement; response-level and perception analysis. PageScoreAccuracy is all-or-nothing by construction because "every response on the page graded correctly" is the deployed requirement; Table 15 handles the page-length dependence (Astra: 62.63% at 1-5 units, 26.38% above 20), Table 13 gives unit accuracy with intervals, Table 12 response-form strata. The new Table 19 adds precision, recall and F1 for the credit class, the two directional rates and the abstention trade-off. On perception, we intervene rather than correlate: C and D supply oracle locations and transcriptions. One dissociation speaks to your concern: the best recogniser among the VLMs (Qwen3.8-Max, EM 33.15%) has a higher false-credit rate (23.8%) than Astra (EM 25.78%, 15.6%), so recognition quality and grading reliability do not move together.
W4 VEHME's three methodological points. Class imbalance: 83.18% credited against 16.82% uncredited per unit, addressed by reporting the two directional rates instead of a single F1, which under that prior is dominated by the majority class (Table 19). Controlled visual input: that is exactly the B-E design, adding one evidence channel at a time under a fixed output contract. Error detection and localisation: covered by the grounding task and its StudentCorrection target, the hardest of the four (best OCR accuracy 1.65%), though we frame it as query-driven localisation rather than in-solution error detection.
Q1-Q5. Q1: the three axes above. Q2: linked identities plus same-page cases, B-E interventions, and the directional decomposition. Q3: Tables 15, 13 and the new Table 19, with the caveat that F1 is prior-inflated. Q4: complete prompts and schemas in App. C.1-C.4, attempt and selection rules in App. B.3; the guideline and an inference-configuration table will be added in the revision. Q5: four full-resolution pages in App. A and nine traceable cases in Figs. 5-13, with the underlying records released.