- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces FCMBench, a multimodal benchmark for financial credit evidence review with 4,922 captured images and 11,045 VQA instances over 26 Chinese and English certificate types.
Critical strengths
S1: This paper builds data around linked documents of the same applicant, which enables cross-document tasks such as CC and RR that general document benchmarks lack. Quality control is careful.
S2: The evaluation design is thorough.
Critical weaknesses
W1: This paper calls three properties "critical" to credit review without evidence beyond 20 interviews, and claims high discriminative power although its own paired bootstrap cannot separate the top three models (Appendix E.4).
W2: This paper calls all nine conditions acquisition artifacts, but watermarks and crops are digital edits of Normal Captures (Appendix A.2). All 11,045 instances come from 36 identities sharing one template per document type.
Questions for authors
Q1: Can the authors compare model scores on the digital source documents and on their physical recaptures? Without this, the claim that digital synthesis misses real capture artifacts remains untested..
Q2: Can the authors report results on a held-out-template or held-out-identity split, and give image-level realism measures against real captures.
Q3: Which logos, emblems or portraits were not replaced? Did the authors check that the fabricated ID numbers, card numbers and AI-generated portraits do not match real individuals?
Q4: Can the authors add OCR-specialized models and an OCR+LLM pipeline baseline, and report NC and RR under higher reasoning effort, to test whether the current ranking still holds?