- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work presents OCR Agent Bench, a benchmark with 260 executable tasks covering seven industrial domains to evaluate agent-level OCR competence beyond plain text recognition, focusing on end-to-end task execution rather than mere recognition accuracy. It also proposes CLOCR Agent, an agent framework incorporating OCR-oriented guidance, source-aware evidence retrieval, structured content access and selective local re-OCR.
Critical strengths
This work features meaningful problem formulation and benchmark design by addressing the gap between traditional OCR evaluation and agent-level document understanding; OCR Agent Bench assesses full task execution and evidence-based artifact generation for practical OCR capability measurement. It also conducts comprehensive empirical evaluations across diverse reasoning models, agent harnesses and OCR backends, with component ablations verifying the necessity of source-addressed tools and OCR-oriented guidance to support the proposed architecture.
Critical weaknesses
-
The improvements specific to OCR cannot be well isolated. End-to-end performance is affected by general reasoning, tool orchestration and artifact generation, while existing experiments fail to separate OCR-related gains from general agent performance, calling for more controlled experiments to validate the core claims.
-
CLOCR Agent only yields inconsistent cross-domain improvements, and higher task completion rates do not guarantee better output quality. Additional statistical analysis and failure case studies are needed to verify its generalizability.
-
Issues exist regarding benchmark reproducibility and evaluation reliability. The benchmark uses agent-generated tasks and automatic evaluation; full human validation of all 260 tasks is unclear, and redistribution constraints hinder independent reproduction. Better documentation on quality control, evaluation variance and benchmark accessibility is required.
Questions for authors
See Weakness.
2 comments
Thank you very much for your time and for reconsidering our score. We greatly appreciate your constructive comments, which have significantly helped improve the quality of our work.
Thank you for your responses, and I will raise my score from 1 to 2.