acceptodds
Under review as a conference paper at ICLR 2027

A Hybrid Approach for Key-Information Extraction from Structured Documents

Abstract

Automated extraction of key information from structured document images is important for accounting, document management, and business automation, yet it remains difficult under blur, noise, mixed languages, and variable layouts. Although recent vision-language-model-based solutions have shown strong performance on document-understanding tasks, many of them rely on large models that require expensive hardware or cloud-based inference. In practical settings, however, document data may be sensitive, and some clients do not permit sending files to external servers, making fully local processing on affordable devices a necessary requirement. This work studies that problem through three extraction settings: a classical pipeline based on field detection, OCR, and post-processing; an OCR-free vision-language pipeline that generates JSON directly from the full document; and a hybrid pipeline that combines field localization with vision-language recognition on cropped regions. The proposed workflow includes orientation correction, field detection, recognition, normalization, and validation for producing machine-readable JSON outputs. Experiments were conducted under a unified schema and included comparisons at the detector, OCR, vision-language-model, and end-to-end levels. The results show that the hybrid pipeline achieved the best overall performance, with lower error rates, higher exact-match scores, and fully valid JSON outputs compared with the other evaluated settings. The study also showed that YOLO11s, Tesseract, and Qwen2-VL-2B-Instruct were the strongest components in their respective comparisons. These findings indicate that combining explicit localization with lightweight vision-language recognition on cropped regions provides a more accurate, practical, and resource-efficient solution for structured receipt understanding than either a fully classical or a fully OCR-free approach.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Hybrid pipeline, key information extraction, optical character recognition, vision-language models, structured documents

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_ocYHOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose a comparative study of 3 different architectures to extract information from structured documents (forms). The proposed architectures are three:

i) one based on object detector (YOLO), which they use to detect text regions before applying an OCR;
ii) one based on a VLM that extracts the information directly from the image, without using OCRs; and

iii) a hybrid one that combines the two previous architectures.

The experiments are carried out on their own datasets and are not compared with reference datasets such as FUNSD (https://guillaumejaume.github.io/FUNSD/)

Critical strengths

None

Critical weaknesses

It essentially is a technical work that combines pre-trained modules. If any model finetuning was done, I was unable to identify it in the manuscript. Moreover, the kind of architectures proposed, at the levei that is done in the paper, does not usually require finetuning

The paper does not provide any relevant scientific contribution. The authors are unaware of what has been published in the field of document analysis in recent years. A simple search by a web search engine with the keywords: "yolo" + "document analysis". It points to a paper, from 2024, (https://arxiv.org/pdf/2410.12628) that presents similar ideas to the “classical” one proposed. In the ICDAR proceedings of 2026 and earlier, one can find works that already propose hybrid schemes based on agency architectures that propose better methods than those proposed by them.

It is necessary to evaluate their proposed architectures in benchmark datasets, e.g FUNSD, IIT-CDIP or RVL-CDIP.

Questions for authors
  • I strongly suggest to the authors making a more exhaustive literature review in the field of Document Image Analysis Recognition and Understanding. In addition to the reference cited above, I recommend reviewing ICDAR’s proceedings of the last years, or specialist journals in the field such as IJDAR.

  • The authors refer to the problem of processing sensitive data or data protected by legislation, but in their evaluation of their methods there are no experiments related to this topic. The development of document analysis methods that preserve data privacy by design is still a hot topic in the document image analysis community

Reviewer_LAKTOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies locally executable receipt information extraction through three pipelines: detector-based OCR, full-document VLM extraction, and a hybrid combining YOLO11s field localization with crop-level VLM recognition and rule-based normalization. Evaluation covers a private receipt collection and a four-field mapping of CORD-v2. The hybrid improves extraction metrics on the private data at increased latency, while CORD-v2 results show model-dependent trade-offs between field accuracy, document exact match, and JSON validity.

Critical strengths

The paper addresses a practical deployment constraint and usefully distinguishes field accuracy, complete-document correctness, JSON validity, and latency. The authors also explicitly acknowledge the negative and statistically nonsignificant CORD-v2 results, providing a balanced account of the limitations of detector-guided recognition rather than presenting cropping as universally beneficial.

Critical weaknesses

Limited methodological novelty and empirical insight. The proposed approach combines an existing detector, pretrained VLMs, field-specific prompts, and standard postprocessing (§3). The paper does not establish a distinct technical advance, and its empirical analysis does not explain sufficiently when or why this combination helps. On CORD-v2, the Qwen2 hybrid underperforms full-document extraction; with Qwen2.5, the item-exact-match improvement is nonsignificant ((p=0.50)), while document exact match decreases from 29% to 18% (Table 3). Although these observations are useful, the current study provides insufficient methodological depth or generalizable empirical insight to constitute a substantial research contribution.

Insufficiently controlled and reproducible evaluation. The exact private end-to-end test set and its relationship to detector training are unclear, while essential details of line-item grouping, prediction–ground-truth matching, and invalid-JSON handling are missing (§§3.2–4.1). The hybrid/full-document comparison also changes task supervision, effective visual input budget, prompting, and JSON assembly without isolating their effects. Stronger local OCR/document-parsing baselines and complete hardware and end-to-end cost reporting are absent. These gaps make it difficult to verify the strongest private-data results or assess whether the proposed design offers a convincing accuracy–cost advantage.

Questions for authors

What are the exact sample counts and split relationships for the private end-to-end evaluation in Table 2? Please specify how detected fields are grouped into line items, how predictions are matched to ground truth, and whether invalid JSON outputs remain in the denominators of all reported metrics.

Reviewer_zeJsOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies key information extraction from receipts for local deployment, comparing three pipelines: detection followed by OCR, full-document extraction using compact vision-language models (VLMs), and detector-guided VLM recognition on cropped regions. The hybrid approach uses YOLO11s to localize fields, invokes a VLM to recognize each field, and then cleans and normalizes the results before assembling them into JSON. The authors conduct component-level and end-to-end evaluations on in-house receipt data and CORD-v2, with the latter mapped to four field categories. The hybrid approach achieves stronger field-level performance on the in-house data.

Critical strengths

The paper addresses the practical need to process sensitive receipts locally and extends its comparison beyond standalone detection or OCR components to pipelines producing structured JSON. The authors separately report field-level, item-level, and document-level correctness, along with JSON validity, showing that improvements in field-level metrics do not necessarily translate into higher document-level success rates. This multilevel evaluation perspective, which considers the complete extraction task, is a strength.

Critical weaknesses
  1. Processing sensitive documents locally is a reasonable practical requirement, but the paper does not adequately explain why this requirement calls for the proposed detection–cropping–per-field VLM architecture.
  2. The method primarily combines off-the-shelf detectors, pretrained VLMs, field-specific prompts, and rule-based post-processing. Equation (1) merely expresses the pipeline as a composition of functions, without clarifying any new learning objective or extraction mechanism.
  3. Although the Related Work section cites recent region-aware and multi-stage extraction methods, it lacks a detailed analysis of how the proposed approach differs from them. It also does not compare against representative complete extraction systems under a common schema and comparable resource budgets.
  4. The paper lacks ablation studies that disentangle the effects of localization, the effective resolution of cropped inputs, field-specific prompting, and programmatic JSON assembly, leaving the sources of the gains unclear.
  5. Comparisons with existing works are missing.
Questions for authors
  1. Exactly how many receipts are included in the in-house test set, and what do the samples look like?
  2. The paper does not spell out CER and WER or provide formal mathematical definitions of these metrics.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.