- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Dianjin-DocBench is a financial document OCR benchmark: 712 documents and 1,033 pages over 16 Chinese and English categories, manually annotated in three stages that separate layout, content recognition and truncated-content linking. It proposes Area F1 for visual-region preservation, aggregates text, formula, table and image scores into an equal-weight Overall score, and evaluates 22 systems.
Critical strengths
S1. The two-level ground truth has real diagnostic value (§3.2–3.3). Keeping the original fragments, the link relations and the correctly merged content side by side separates local recognition failures from document reconstruction failures. Scoring link P/R/F1 apart from reconstructed-text Edit exposes the high-precision, low-recall joining pattern (Tables 3, 5).
S2. The evaluation is broad and internally consistent. 22 systems, most released in 2026, run under one protocol with language, category and scope breakdowns, and Eq. 2 recovers every row of Table 2 to ±0.01. That same-page table joining is essentially unsolved (best F1 12.92, Table 4) is a potentially important negative result.
Critical weaknesses
W1. The headline ranking and the category conclusion follow from the aggregation rule. Overall is an equal-weight mean of four terms in which a system without layout output scores 0 on image, i.e. −25 points (Eq. 2). Across the 22 systems Image F1 alone explains R² ≈ 0.95 of the variance in Overall; on text, formula and table alone the three families are tied (91.11, 91.23, 91.08), and the three-term ranking has Kendall τ ≈ 0.30 with the reported one. So §5.1's claim that pipelines lead page-level performance is a statement about the image term. The same empty-set rule produces the §5.4 "exam-paper advantage": exam pages have essentially no ground-truth images, and a page with no ground truth but one predicted box scores 0, so Exam-Papers lands at 0.72–0.77 of each system's own three-term mean for all 17 specialized systems against 0.96–1.02 for all 5 general VLMs (Table 8).
W2. The added value is not established and the evaluation is under-specified. The paper's own Table 1 shows MPDocBench-Parse covering same-page and cross-page link F1 with 3,246 pages against 1,033 here, and FinDocBench already including financial cross-page tables. Area F1's only evidence is the constructed example in Fig. 4 (77.50 vs. 76.54), which scores both measures over full rectangles, whereas MinerU2.5 computes PageIoU over the page's non-background mask — exactly what makes it tolerant of coarse boxes. The continuation evaluation never defines the matching threshold, what counts as a predicted link, or how spurious merges are scored, and Table 5 implies only 84 cross-page text links. Nothing is reported on release, license, document sourcing, privacy or annotation quality control, no result carries a confidence interval, and 352 of the 1,033 pages are not financial documents.
Questions for authors
- How do Overall and the exam-paper conclusion change if the image term is reported separately, and if Image F1 is computed on a fixed page set with a separate false-positive rate for image-free pages?
- In the continuation evaluation, how are predicted links defined, what matching threshold is used, and how are spurious merges scored?
- Will the data and evaluation code be released, under what license, and what annotation quality-control evidence exists (double-annotation coverage, inter-annotator agreement for truncation links)?
- What does this benchmark add over MPDocBench-Parse and FinDocBench, and is there evidence on real predictions that Area F1 is preferable to PageIoU on the non-background mask?
Response to the Authors
Thank you for the rebuttal. However, it does not fully resolve my concerns regarding the validation and justification of the proposed evaluation metric. Therefore, I maintain my original score.