acceptodds
Under review as a conference paper at ICLR 2027

Dianjin-DocBench: A Full-Spectrum Benchmark for Financial Document OCR

Abstract

Financial document parsing must preserve both the content of individual pages and the continuity of information fragmented by layout and pagination. We introduce Dianjin-DocBench, a benchmark covering 712 documents and 1,033 pages across 16 financial and finance-related categories in Chinese and English. Its manual annotation workflow separates layout analysis, content recognition, and truncated-content linking, producing page-level ground truth with unmerged regions and document-level ground truth with explicit relations and correctly consolidated text and tables. The evaluation protocol covers text, formula, table, and image parsing, alongside separate same-page and cross-page continuation tasks. We propose a new image evaluation metric that measures overlap between unions of predicted and annotated regions, accommodating differences in region granularity while penalizing both missed visual content and excess predicted area. We evaluate pipeline-based specialized VLMs, end-to-end specialized VLMs, and general VLMs under this unified protocol. By separating recognition, visual preservation, and continuation, Dianjin-DocBench provides a focused evaluation of the information retained in parsed financial documents.

Primary area:
datasets and benchmarks
Keywords:
OCR, Benchmark, Financial Document Parsing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XGffOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Dianjin-DocBench is a financial document OCR benchmark: 712 documents and 1,033 pages over 16 Chinese and English categories, manually annotated in three stages that separate layout, content recognition and truncated-content linking. It proposes Area F1 for visual-region preservation, aggregates text, formula, table and image scores into an equal-weight Overall score, and evaluates 22 systems.

Critical strengths

S1. The two-level ground truth has real diagnostic value (§3.2–3.3). Keeping the original fragments, the link relations and the correctly merged content side by side separates local recognition failures from document reconstruction failures. Scoring link P/R/F1 apart from reconstructed-text Edit exposes the high-precision, low-recall joining pattern (Tables 3, 5).

S2. The evaluation is broad and internally consistent. 22 systems, most released in 2026, run under one protocol with language, category and scope breakdowns, and Eq. 2 recovers every row of Table 2 to ±0.01. That same-page table joining is essentially unsolved (best F1 12.92, Table 4) is a potentially important negative result.

Critical weaknesses

W1. The headline ranking and the category conclusion follow from the aggregation rule. Overall is an equal-weight mean of four terms in which a system without layout output scores 0 on image, i.e. −25 points (Eq. 2). Across the 22 systems Image F1 alone explains R² ≈ 0.95 of the variance in Overall; on text, formula and table alone the three families are tied (91.11, 91.23, 91.08), and the three-term ranking has Kendall τ ≈ 0.30 with the reported one. So §5.1's claim that pipelines lead page-level performance is a statement about the image term. The same empty-set rule produces the §5.4 "exam-paper advantage": exam pages have essentially no ground-truth images, and a page with no ground truth but one predicted box scores 0, so Exam-Papers lands at 0.72–0.77 of each system's own three-term mean for all 17 specialized systems against 0.96–1.02 for all 5 general VLMs (Table 8).

W2. The added value is not established and the evaluation is under-specified. The paper's own Table 1 shows MPDocBench-Parse covering same-page and cross-page link F1 with 3,246 pages against 1,033 here, and FinDocBench already including financial cross-page tables. Area F1's only evidence is the constructed example in Fig. 4 (77.50 vs. 76.54), which scores both measures over full rectangles, whereas MinerU2.5 computes PageIoU over the page's non-background mask — exactly what makes it tolerant of coarse boxes. The continuation evaluation never defines the matching threshold, what counts as a predicted link, or how spurious merges are scored, and Table 5 implies only 84 cross-page text links. Nothing is reported on release, license, document sourcing, privacy or annotation quality control, no result carries a confidence interval, and 352 of the 1,033 pages are not financial documents.

Questions for authors
  1. How do Overall and the exam-paper conclusion change if the image term is reported separately, and if Image F1 is computed on a fixed page set with a separate false-positive rate for image-free pages?
  2. In the continuation evaluation, how are predicted links defined, what matching threshold is used, and how are spurious merges scored?
  3. Will the data and evaluation code be released, under what license, and what annotation quality-control evidence exists (double-annotation coverage, inter-annotator agreement for truncation links)?
  4. What does this benchmark add over MPDocBench-Parse and FinDocBench, and is there evidence on real predictions that Area F1 is preferable to PageIoU on the non-background mask?
Reviewer_GmjuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Dianjin-DocBench, a benchmark containing 712 documents and 1,033 pages across 16 financial and finance-related categories in Chinese and English. It provides page-level content annotations and document-level continuation relations for text and tables, together with an Area F1 metric for visual-region preservation. Experiments evaluate 22 specialized and general vision-language models, distinguishing recognition quality from successful content reconstruction and revealing substantial weaknesses in within-page table merging.

Critical strengths

The paired page-level and document-level annotations provide a useful basis for distinguishing recognition errors from continuation failures (Sections 3.3 and 4.3). This distinction matters for practical document parsing, where accurate transcription alone does not ensure a coherent output. The separate content and connection metrics are informative: Table 4 shows that moderate table-similarity scores can coexist with near-zero joining F1, exposing failures that content-level evaluation alone could obscure.

Critical weaknesses

The incremental contribution over closely related benchmarks is insufficiently established. Table 1 shows that MPDocBench-Parse already covers figure evaluation and truncated-text/table reconstruction, while FinDocBench addresses financial-document parsing and truncated-table reconstruction. The paper does not sufficiently demonstrate which distinctive financial challenges its dataset adds. Moreover, Area F1 is mathematically Dice overlap over unions of bounding-box rectangles (Equation 1). This may be a useful evaluation choice, but the illustrative comparison in Figure 4 is not complemented by empirical comparisons with existing visual metrics or validation against human judgments. Its additional diagnostic value therefore remains unclear.

The benchmark’s reliability and reproducibility are insufficiently documented. Sections 3–5 provide limited information about document sourcing and sampling, annotation verification, and the numbers and lengths of continuation groups. Section 4.3 also leaves the recognition-tolerant matching procedure and the enumeration of false joins underspecified, while prompts and multi-page inference configurations are not reported. These details directly affect the interpretation of continuation scores and whether comparisons across systems can be reproduced, making their omission consequential for a benchmark contribution.

Questions for authors

The following are clarification requests that could materially affect my assessment. What empirical evidence demonstrates that this dataset and Area F1 reveal failures or change conclusions beyond existing benchmarks and visual metrics? Do the 712 documents represent complete source documents or selected page sequences, and what annotation verification procedures and continuation-group statistics support the benchmark’s reliability? Can the authors specify the matching and false-join rules, model eligibility criteria, prompts, and multi-page input configurations needed to reproduce the evaluation?

Reviewer_wA3LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors introduce Dianjin-DocBench which is a benchmark covering 712 documents and 1,033 pages across 16 financial and finance-related categories in Chinese and English. For the evaluation, the authors introduce Area F1 and compare 22 document parsing systems. It evaluates content recognition, visual preservation, and same-page and cross-page continuation.

Critical strengths
  • Separating recognition from truncated-content reconstruction reveals failures that transcription scores can conceal. The low same-page table-linking scores in Table 4 show why this distinction matters.
  • Area F1 handles overlapping visual regions and differences in region granularity. The language and document-category breakdowns make the benchmark more informative than a single aggregate ranking.
Critical weaknesses
  • Section 3.2 describes the annotation stages. The contribution is mostly annotation, yet the paper gives no number of annotators. It does not explain how annotation errors and disagreements were checked or resolved. The counts of same-page and cross-page continuation groups are also missing. These details are important for judging the reliability of the benchmark's distinctive tasks.

  • The paper also does not say where the documents come from, under what licenses, or whether they will be released.

  • Section 4.3 leaves recognition-tolerant matching underspecified. Matching thresholds and the handling of unmatched fragments can affect link scores, yet they are not given. Prompts and document-input settings are also needed to reproduce the comparisons between general vision-language models and specialized systems.

  • The case against PageIoU is a single hand-made figure (Figure 4). The paper does not show how often the overlap penalty occurs in real predictions, whether the 22-system ranking changes under PageIoU-F1 or a standard IoU-matched F1, or whether Area F1 tracks human judgment.

Questions for authors

Please address the points raised in the critical weaknesses section.

1 comment
Reviewer_wA3LOpenReview ↗

Response to the Authors

Thank you for the rebuttal. However, it does not fully resolve my concerns regarding the validation and justification of the proposed evaluation metric. Therefore, I maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.