acceptodds
Under review as a conference paper at ICLR 2027

Agent-Level OCR: A Benchmark and An OCR-Enhanced Agent

11 views

Abstract

Optical character recognition (OCR) has advanced from transcribing individual characters to recognition and layout-aware parsing, supported by increasingly capable models and benchmarks. Existing benchmarks, however, primarily evaluate a model response to a prescribed image or document. A complete agent faces a broader problem: it must recognize when OCR is needed, locate the relevant visual evidence, verify uncertain observations, and carry the recovered information to complete a task. We define this ability to acquire, verify, and use visual-text information during task execution as agent-level OCR capability. To measure it, we introduce OCR Agent Bench, which extends the evaluation unit from an isolated model response to a complete agent execution spanning the reasoning model, harness, skills, tools, and OCR backend. The benchmark contains 260 executable tasks that evaluate delivered artifacts against task-specific criteria and source evidence. This evaluation target also motivates CLOCR Agent. Pre-extracted OCR alone does not tell an agent where to inspect, when to obtain another observation, or how to associate recovered information with its source. CLOCR Agent instead turns OCR from fixed preprocessing into an on-demand, locally verifiable, and source-associated capability through OCR-oriented guidance, source-addressed evidence operations, structured content access, and selective local OCR. Across 17 evaluated model–harness configurations, CLOCR Agent achieves the highest overall score (68.3) and ranks first for three of the four reasoning models.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Agent Benchmark, OCR Agent

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_r91kOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work presents OCR Agent Bench, a benchmark with 260 executable tasks covering seven industrial domains to evaluate agent-level OCR competence beyond plain text recognition, focusing on end-to-end task execution rather than mere recognition accuracy. It also proposes CLOCR Agent, an agent framework incorporating OCR-oriented guidance, source-aware evidence retrieval, structured content access and selective local re-OCR.

Critical strengths

This work features meaningful problem formulation and benchmark design by addressing the gap between traditional OCR evaluation and agent-level document understanding; OCR Agent Bench assesses full task execution and evidence-based artifact generation for practical OCR capability measurement. It also conducts comprehensive empirical evaluations across diverse reasoning models, agent harnesses and OCR backends, with component ablations verifying the necessity of source-addressed tools and OCR-oriented guidance to support the proposed architecture.

Critical weaknesses
  1. The improvements specific to OCR cannot be well isolated. End-to-end performance is affected by general reasoning, tool orchestration and artifact generation, while existing experiments fail to separate OCR-related gains from general agent performance, calling for more controlled experiments to validate the core claims.

  2. CLOCR Agent only yields inconsistent cross-domain improvements, and higher task completion rates do not guarantee better output quality. Additional statistical analysis and failure case studies are needed to verify its generalizability.

  3. Issues exist regarding benchmark reproducibility and evaluation reliability. The benchmark uses agent-generated tasks and automatic evaluation; full human validation of all 260 tasks is unclear, and redistribution constraints hinder independent reproduction. Better documentation on quality control, evaluation variance and benchmark accessibility is required.

Questions for authors

See Weakness.

2 comments

Thank you for your responses, and I will raise my score from 1 to 2.

Thank you very much for your time and for reconsidering our score. We greatly appreciate your constructive comments, which have significantly helped improve the quality of our work.

Reviewer_S8fYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies agent-level OCR: the ability to acquire, verify, and use visually encoded information to complete tasks. It introduces OCR Agent Bench, comprising 260 executable tasks across seven industry domains, which evaluates delivered artifacts against task requirements and source evidence. The paper also proposes CLOCR Agent, combining an OCR-oriented skill, source-addressed Search and Read operations, and selective region-level re-OCR. Experiments compare 17 model–harness configurations and include OCR-backend, component, and execution-budget ablations. CLOCR achieves the highest reported Overall score of 68.3 and ranks first for three of the four matched reasoning models, with gains varying across models and domains.

Critical strengths

The paper addresses a worthwhile research problem with practical relevance. The experiments cover multiple models, component ablations, and failure cases, and the authors explicitly acknowledge that the gains vary across models and domains.

Critical weaknesses

My biggest concerns are: 1,Insufficient explanation of model–harness interactions. Table 1 shows substantial performance differences across harnesses using the same reasoning model. For example, Qwen3.8-Flash achieves Overall scores of 9.9 with Codex and 57.7 with Hermes, with completion rates of 12.7% and 79.2%, respectively. The relative ranking also changes across models: CLOCR outperforms Codex with DeepSeek but underperforms it with GLM. Although Figure 6 reports failure composition, the analysis does not sufficiently explain these interactions. The time-budget ablation covers only CLOCR with GLM. Consequently, it remains unclear how much of the observed variation reflects evidence-handling capabilities versus compatibility, execution policies, or latency, limiting the generalizable insights from the comparison. 2, Limited isolation of the contribution beyond conventional OCR-based pipelines. Table 4 compares OCR backends, and Table 5 includes a pre-extracted OCR baseline, providing useful initial evidence. However, the latter comparison covers only GLM, and its implementation is insufficiently specified to establish whether it preserves comparable document structure, source references, and access to relevant content.

Questions for authors

What explains the largest model–harness performance gaps, particularly Qwen3.8-Flash with Codex versus Hermes, and the reversal between Codex and CLOCR across GLM and DeepSeek? Could you provide a focused analysis of representative failures distinguishing compatibility or tool-interface issues, timeouts, and task-level reasoning or evidence-use errors?

Please clarify the visual-input and tool configuration of each model–harness combination. Which reasoning models support image inputs, and did each harness actually pass source images, rendered pages, or image crops to those models? Which configurations instead relied on extracted text or external OCR services? Without this information, it is difficult to distinguish the effects of native visual capabilities, OCR backend access, and harness orchestration on the reported performance differences.

1 comment
AuthorsOpenReview ↗

A Friendly Follow-up on Our Rebuttal and Additional Evidence

Dear Reviewer S8fY,

Thank you again for your time and careful review. We would like to follow up on our responses and the additional evidence provided to address your concerns.

Our responses include new tool-removal experiments, time-budget comparisons across additional model–harness configurations, and paired analyses separating completion coverage from output quality. We have also clarified the visual-input pathways, tool configurations, and evaluation procedure.

When you have a chance, we would appreciate your feedback on whether these additions address your main concerns and which points, if any, still require clarification. We would be happy to provide further details during the discussion. If the additional evidence changes your view of the work, we would also appreciate your consideration of updating your assessment.

Thank you for your time and consideration.

Best regards, The Authors

Reviewer_ULsgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces OCR Agent Bench, a benchmark containing 260 executable tasks across seven industry domains. It evaluates how agents acquire, verify, and use visual-text evidence to complete tasks and produce the required artifacts. The authors also propose CLOCR Agent, which combines OCR-oriented guidance, source-addressed tools, structured content access, and selective local re-OCR. The evaluation covers 17 model–harness configurations, with the best configuration achieving an Overall score of 68.3.

Critical strengths

S1. Relevant problem formulation and benchmark design. The paper evaluates OCR as part of complete agent tasks involving evidence acquisition, verification, and artifact generation. The 260 tasks cover diverse document types and practical requirements, making the benchmark relevant to real document-processing workflows. S2. Broad experimental evaluation. The authors evaluate 17 model–harness configurations and compare task performance and resource consumption. The component and execution-time experiments also provide useful evidence about the effects of OCR tool access and task completion reliability.

Critical weaknesses

W1. Insufficient validation of the benchmark's reliability and diagnostic value. OCR Agent Bench uses Agent-assisted task construction and LLM-based judging. The paper does not report how many of the 260 tasks underwent independent human validation, or specify the sample size and selection procedure for the Judge–human agreement study. These details are important for assessing the reliability of the benchmark. The evaluation covers artifact completion and several quality dimensions, but does not systematically distinguish OCR recognition, evidence localization, and verification errors from general execution failures. The benchmark's additional diagnostic value over existing document-agent benchmarks remains insufficiently established. W2. The contributions of the proposed agent mechanisms are not sufficiently isolated. CLOCR Agent combines source-addressed evidence, structured content access, and selective local re-OCR. These mechanisms are closely related to capabilities already explored in document-agent systems, making their individual contributions important to establish. Table 5 evaluates the OCR skill and tool access as broader components, including a −Skill configuration that retains OCR tools. The experiments do not separately measure the effects of source addressing, structured content access, or local re-OCR. It is therefore difficult to determine which specific design choices account for the improvements. W3. The sources of the reported performance gains are not sufficiently explained. With DeepSeek-V4.1-Flash, CLOCR Agent improves Overall from 60.5 to 68.3 over Claude Code, alongside a Completion increase from 84.6% to 95.4%. Conditional Quality remains nearly unchanged at 71.5 and 71.6, although these values are calculated over different sets of completed tasks. Other reasoning models show different patterns. These results indicate that task completion reliability is an important contributor to performance. The paper does not provide a systematic breakdown of OCR recognition errors, evidence-handling errors, and execution failures to identify the improvements in OCR-specific capabilities. Each task is also evaluated through a single execution, leaving smaller performance differences without sufficient statistical support.

Questions for authors

Q1. Could the authors conduct an independent human audit of a representative set of benchmark tasks and report the validation results by task or challenge category? Please also provide the sample size and selection procedure for the Judge–human agreement study and examine whether the three Judges produce consistent system rankings.

Q2. Could the authors extend the component analysis with separate ablations for source addressing, structured content access, and local re-OCR, while keeping the remaining tools and execution settings unchanged? This would clarify the contribution of each mechanism to the full system.

Q3. Could the authors provide a task-level failure analysis separating OCR recognition errors, evidence-handling errors, and execution failures? A paired quality comparison on tasks completed by both methods, together with confidence intervals for the main performance differences, would help establish the sources and reliability of the reported gains.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal Updates and Additional Evidence

Dear Reviewers and Area Chair,

Thank you for your careful evaluation and constructive feedback. During the rebuttal, we added experiments and analyses addressing the contributions of OCR tools, execution budgets, statistical support, and evaluation settings. We summarize the main updates below to make the additional evidence easier to review.

  1. Contributions of individual OCR tools and pre-extracted content.

We added three tool-removal experiments, each covering all 260 tasks with GLM. Removing Search, Read, or local re-OCR yields Overall scores of 30.9, 31.6, and 38.3, respectively, compared with 50.8 for CLOCR Agent. On tasks completed by both configurations, the corresponding Quality scores decrease by 3.42, 1.32, and 1.81 points. This comparison shows that the differences extend beyond completion rates to the quality of outputs on matched tasks.

We also clarified the static-OCR baseline: supplying pre-extracted OCR raises Overall from 37.2 to 44.0, while CLOCR Agent reaches 50.8. These results distinguish the benefit of supplying recognition content from that of providing interactive evidence access and local reinspection (Table 5; Appendix A.3.4).

  1. Execution-budget analysis across additional harnesses.

We extended the budget study to Qwen3.8-Flash with Codex and Hermes, running all 260 tasks at both 800 and 1200 seconds. Codex’s Overall increases from 30.7 to 45.8, with completed tasks increasing from 111 to 160. Hermes’s Overall increases from 62.5 to 65.4, with completions increasing from 240 to 257.

The Hermes–Codex Overall gap narrows from 31.83 to 19.56 points. Paired analysis attributes 14.10 points of Codex’s 15.12-point gain to additional completion coverage. These results connect the observed budget differences to successful artifact delivery and highlight timely completion as an important component of agent performance (Table 6; Appendix A.3.5).

  1. Statistical support and sensitivity to Judge choice.

We added 11 paired comparisons that separate completion coverage from quality on jointly completed tasks, together with confidence intervals from 20,000 paired task-bootstrap samples. DeepSeek with CLOCR Agent exceeds DeepSeek with Claude Code by 7.79 Overall points, with a 95% interval of [3.51, 12.01].

We also recomputed system rankings separately under each of the three LLM Judges. The rankings show strong agreement, with Kendall’s W = 0.9897. These analyses provide statistical context for the reported differences and show how aggregate rankings vary with Judge choice (Appendices A.3.1–A.3.2).

  1. Clearer benchmark construction, visual inputs, and evaluation procedures.

We clarified the ADE construction process and subsequent human screening, the visual-input pathways used by the harnesses, and the shared six-dimensional scoring protocol. The revised description explains how agents select files, pages, or regions through tools and how the resulting visual evidence enters subsequent model requests. It also specifies how task requirements, reference evidence, execution traces, and delivered artifacts enter evaluation (Section 4.1; Appendices A.5–A.6).

Expanded domain and case analyses further explain differences in evidence acquisition, source-to-field binding, verification, and final delivery. Our broader objective is to evaluate how complete agents acquire and use visual-text evidence to produce grounded task artifacts within practical resource budgets.

We would appreciate the reviewers’ feedback on these additions, particularly whether any concerns remain unresolved or would benefit from further clarification. We hope the additional evidence will inform your final assessment, and we remain available to respond during the discussion.

Thank you again for your time and consideration.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.