acceptodds
Under review as a conference paper at ICLR 2027

FCMBench: A Large-scale Financial Credit Multimodal Benchmark for Real-world Applications

Abstract

We introduce FCMBench, a large-scale multimodal benchmark designed for understanding identity-linked documents in financial credit evidence review applications. It contains 4,922 real-captured images and 11,045 VQA instances covering 26 certificate types, three perception tasks, four credit-specific reasoning tasks, and ten real-world robustness challenges. To address the privacy–realism trade-off in financial data construction, we develop a synthetic-to-physical pipeline that creates linked fictional applicant profiles, generates certificate templates, physically fabricates documents, and recaptures them under controlled scenarios. Task design is grounded in interviews with senior credit reviewers, and a two-stage audit refines data quality. We evaluate 39 proprietary and open-source vision–language models from 14 organizations. Results show substantial performance gaps among current VLMs, with overall F1 scores ranging from 24.60 to 71.57 (mean 50.9; SD 12.6 across models). Matched robustness evaluations further reveal consistent degradation under challenging conditions, particularly for multi-document compositions, small regions of interest, and defocus. Extensive experimental results demonstrate that FCMBench is challenging and highly discriminative for identifying performance gaps among vision–language models.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Benchmark, Financial Credit, Real-world Applications

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_U5XiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FCMBench, a multimodal benchmark for financial credit evidence review with 4,922 captured images and 11,045 VQA instances over 26 Chinese and English certificate types.

Critical strengths

S1: This paper builds data around linked documents of the same applicant, which enables cross-document tasks such as CC and RR that general document benchmarks lack. Quality control is careful.

S2: The evaluation design is thorough.

Critical weaknesses

W1: This paper calls three properties "critical" to credit review without evidence beyond 20 interviews, and claims high discriminative power although its own paired bootstrap cannot separate the top three models (Appendix E.4).

W2: This paper calls all nine conditions acquisition artifacts, but watermarks and crops are digital edits of Normal Captures (Appendix A.2). All 11,045 instances come from 36 identities sharing one template per document type.

Questions for authors

Q1: Can the authors compare model scores on the digital source documents and on their physical recaptures? Without this, the claim that digital synthesis misses real capture artifacts remains untested..

Q2: Can the authors report results on a held-out-template or held-out-identity split, and give image-level realism measures against real captures.

Q3: Which logos, emblems or portraits were not replaced? Did the authors check that the fabricated ID numbers, card numbers and AI-generated portraits do not match real individuals?

Q4: Can the authors add OCR-specialized models and an OCR+LLM pipeline baseline, and report NC and RR under higher reasoning effort, to test whether the current ranking still holds?

Reviewer_A82POpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces FCMBench, a multimodal benchmark for evaluating vision-language models in financial credit document understanding. The benchmark comprises 4,922 physically captured document images and 11,045 VQA instances covering 26 document types, seven perception and reasoning tasks, and ten challenging input conditions. To balance privacy and realism, the authors propose a synthetic-to-physical construction pipeline that generates fictional applicant profiles, fabricates physical documents, and captures them under diverse conditions. The paper evaluates 39 vision-language models and provides additional analyses involving human experts, robustness challenges, and transfer to confidential real-world credit documents. The results reveal substantial performance differences among current models and highlight persistent difficulties in fine-grained information extraction and multi-document reasoning.

Critical strengths
  1. Practical and meaningful benchmark setting. The paper addresses an important application of multimodal models in financial credit review. Unlike benchmarks focusing on isolated OCR or general document understanding, FCMBench integrates document recognition, information extraction, cross-document verification, and numerical reasoning within a unified evaluation framework.

  2. Thoughtful dataset construction. The synthetic-to-physical pipeline is a useful approach to balancing privacy protection and realistic image acquisition. The inclusion of diverse capture conditions, multiple document types, and identity-linked evidence provides a reasonably comprehensive evaluation setting.

  3. Extensive experimental evaluation. The evaluation of 39 VLMs offers broad coverage of current models. The inclusion of independent human evaluation, real-world transfer validation, answerability auditing, and identity-clustered bootstrap analysis further strengthens the empirical study.

Critical weaknesses
  1. Limited diversity of underlying identities and document templates. Although FCMBench contains 11,045 VQA instances, Appendix A.1 indicates that these samples are constructed from only 36 fictional applicant identities, with each document type sharing a common template structure. Consequently, many questions are correlated through repeated identities, layouts, and document content. While the authors appropriately account for identity-level correlations in their bootstrap analysis, the limited diversity may still restrict the benchmark's ability to assess generalization to unseen document layouts and applicant profiles.

  2. Potential confounding factors in robustness evaluation. The robustness comparisons match fine-grained task categories rather than identical questions or document instances, as acknowledged in Appendix E.2. Therefore, differences in applicant identities, prompts, and sample distributions may influence the reported performance changes. Furthermore, the Image Quality Evaluation task directly evaluates the presence of capture artifacts, potentially confounding the measurement of robustness degradation. A strictly paired evaluation on identical document-question instances, preferably excluding IQE, would provide stronger evidence.

  3. Limited scope of real-world transfer validation. Although the reported correlation between FCMBench and real-world performance is encouraging, the transfer experiment covers only nine selected models and 385 prompts from 356 confidential document images. The relatively small evaluation set and its restricted accessibility limit independent verification and the generality of the conclusions. Additional validation across more document sources and model families would strengthen the claim that FCMBench reliably predicts real-world performance.

  4. Lack of comparisons with specialized document-processing pipelines. The benchmark evaluates a wide range of general-purpose VLMs, but does not include dedicated OCR-based extraction pipelines combined with structured validation, numerical computation, or rule-based document checking. Such approaches are relevant to financial credit review and could provide useful reference points for determining whether the observed errors arise primarily from visual perception, document understanding, or reasoning limitations.

  5. Sensitivity of the aggregate metric to task composition. The Overall F1 assigns equal weight to all seven tasks despite substantial differences in their sample sizes, such as 4,506 DTR instances versus only 252 RR instances. Moreover, the relatively high random-answer baselines on VC and RR indicate that some tasks have considerable label-prior effects. Although the authors provide chance-corrected ranking analysis, additional evaluations under alternative task-weighting strategies would help establish the robustness of the reported model comparisons.

Questions for authors

Given that FCMBench is constructed from only 36 fictional identities and shared document-type templates, how well do the evaluated models generalize to previously unseen applicant profiles and document layouts? Could the authors provide a held-out template or identity generalization analysis?

Could the authors provide robustness comparisons using identical document-question instances across normal and degraded conditions, excluding Image Quality Evaluation? Would the main robustness conclusions remain unchanged?

How representative is the confidential real-world evaluation set in terms of document diversity and task coverage? Could the authors report uncertainty estimates for the observed rank correlation or provide further validation across additional document sources?

Have the authors considered incorporating OCR-based document-processing pipelines with deterministic numerical calculations and domain-specific validation rules as additional baselines? How would these approaches compare with the evaluated VLMs?

Reviewer_AkGCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FCMBench is a financial document evaluation benchmark that stress test how VLMs review domain-specific task without using real customer data. They whitelist the data by creating fake applicant identities across 26 document types, printing them out as physical copies, and photographing using 5 snapshot setups with 9 environmental conditions. The benchmark has 11k questions on 4.9k images across 39 models being splitted into perception and reasoning heavy tasks based on the interviews with credit reviewers. Human expert reach 91.98 F1 vs 71.57 for top models, which they also show model ranking match performance on a private real-document set high correlation (p = 0.93).

Critical strengths
  1. Rigorous data creation pipeline that is privacy-preserving. They managed to capture the realism of private document capture via real-world printing captured by real phones to transfer the photographic features genuity (blur, glare, angles, etc.). Linking documents per profile also keeps the multi-document tasks realistic (App. A1) and ablation results show image context really matters since text-only drops heavily on perception result (Table 3).

  2. Careful and significant evaluation setup. They do bootstrapping instead treating all questions as independent (App. E3) and include baselines such as random, constant, and modality dropping (text-only) (Table 3) alongside human expert checks (section 4.2.3) + real document validation (section 4.2.4). This approximates careful evaluation especially within domain-specific treatment like this financial case.

Critical weaknesses
  1. Low layout diversity despite the large image count. FCMBench pooled only from the initial 36 seeds of fake people with just one single template per document type (section 3.2, App. A1), and they reuse those exact same physical printouts across all test conditions (App. A2). That means each document type only ever has one layout, which isn't realistic since real bank statements or IDs look totally different from, lets say, bank to bank. Perception perturbation / variations could bring much better evaluation nuances here rather than fixated across one control.

  2. Scores validity and task weighting. On tasks like VC and RR, simple random guessing beats almost half the models (App. E8), and the lowest overall score of 24.60 is below random chance (32.56, Table 3). The authors show the ranking holds after chance correction (App. E8), but this makes overall range makes it questionable on how well the benchmark separates performance It’s also unclear whether weak models is weak because of reasoning or JSON formatting errors, which score zero (Sec. 3.5).

Questions for authors
  1. It would be great if the author can show evaluation result on unseen layouts or multi-issuer templates to showcase that the benchmark does not measure layout memorization.

  2. Would chance-corrected or count-weighted overall scores change the model rankings? Reason -> equal weighting gives full weight to subtasks like VC and RR where random guessing beats half the models (App. E.8), so adjusting for chance would remove floor-effect noise.

  3. What are the correct numbers for the discrepancies between text and appendix? Reason -> Figure 21 states p = 0.35 and 0.46 while App. E.6 says 0.27 and 0.42$ Table 2 image counts sum to 5,416 instead of 4,922, and text says "over 20" reviewers (section 1, and 3.1) vs exactly 20 in Table 6.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.