- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces FactID, a dataset of 78,702 generated Aadhaar documents containing template- and content-level inconsistencies. Evaluations of pretrained forensic models, vision-language models, an OCR-based baseline, and human participants highlight limitations in detecting logical errors without conventional visual artifacts. The problem is relevant, but limited coverage and unresolved evaluation issues weaken the contribution.
Critical strengths
-
The paper addresses a meaningful problem: methods that detect visual artifacts such as splicing, face manipulation, and font irregularities may overlook documents containing errors introduced during generation. Incorporating field validation and cross-field consistency checks into document verification has practical value.
-
FactID contains 78,702 samples covering template- and content-level errors alongside different face types. Each fake document contains one logical error, enabling clear annotations and targeted analysis of detection failures.
-
The controlled experiments demonstrate the limited transferability of existing models. When genuine faces are retained and only logical errors are introduced, the 13 forensic models achieve AUCs of 48.78%–52.55%, indicating that these pretrained models cannot directly handle such errors. Separate evaluations of full documents, faces, templates, and content fields provide further diagnostic insight.
Critical weaknesses
-
FactID contains only Indian Aadhaar cards generated from templates and predefined rules. Without evaluation across countries, document types, or templates, its large sample size does not establish diversity or generalizability.
-
The benchmark covers only part of document authentication: consistent content does not imply authenticity, and genuine documents may contain clerical errors. It lacks systematic coverage of internally consistent forgeries, text–QR discrepancies, capture degradation, legitimate template variations, and compound or unseen errors.
-
Most evaluated models target visual tampering and are tested without adaptation, so their failure does not establish the task’s difficulty. The OCR-plus-rules baseline lacks implementation details and error analysis. The concern is not the absence of a new model, but insufficient evidence of the benchmark’s difficulty and research value.
-
Some results require clarification. ADCD-Net’s aggregate Content Only AUC is 84.32%, while its three category-level AUCs range from 52.17% to 58.69%; this is inconsistent if the same genuine samples and scoring procedure are used. GradCAM alone also cannot establish that 100% AUC reflects “decision collapse.”
-
Names and facial appearance cannot reliably determine gender, making the “gender mismatch” label questionable. Unadapted-model failures demonstrate limited transfer, while the small human study does not justify claims that logical errors are “invisible to humans.”
Questions for authors
- How many templates are represented, and can generalization be tested on another document type or unseen template?
- How is gender mismatch defined, and is the required evidence observable in the document?
Besides, please answer the questions listed in weakness.
2 comments
3. Implementation details of OCR baselines:
We implement a training-free, rule-based verification pipeline using EasyOCR, configured for English and Hindi text recognition. The OCR engine extracts textual information directly from document images, followed by regular-expression-based field extraction and symbolic validation. The evaluation is conducted under three strategies: Full ID, Template Only, and Content Only, using the corresponding test sets. The extracted information is evaluated using three categories of validation rules:
Aadhaar Number Validation: The extracted identifier is checked for its 12-digit structure and mathematical validity using the Verhoeff checksum algorithm. Missing identifiers receive a lower penalty to account for potential OCR extraction failures.
Date Consistency Validation: Extracted dates are parsed using the DD/MM/YYYY format and checked against chronological constraints, including whether the date of birth precedes the issue date, whether the issue date is not in the future, and whether the calculated age falls within a plausible range (0–120 years).
Template Validation: Fixed institutional information, including authority names, official website information, and predefined spelling variations, is verified using string matching and fuzzy similarity through Python's SequenceMatcher, with a similarity threshold of 0.8 for principal template strings.
Each detected violation contributes a predefined penalty, and the accumulated violation score is compared against a fixed threshold of 0.4 to classify the document as genuine or manipulated. The continuous violation score is used to compute AUC.
4. Justification of 100% AUC and gender mismatch:
We agree that 100% AUC and 0% EER indicate perfect score-based discrimination on the evaluated samples, and we clarify the interpretation in Appendix A.5–A.6. However, perfect discrimination does not necessarily imply that a model identifies the underlying logical manipulation, as predictions may rely on incidental visual cues or spurious correlations. This distinction is also illustrated in Figure 1, Such cases demonstrate that correct classification does not necessarily reflect an understanding of genuine forgery, leading to "decision collapse".
Regarding gender mismatch, , rather than inferred solely from facial appearance or names. Each identity document is associated with an assigned demographic profile, which serves as the reference for evaluating gender consistency. A gender mismatch is introduced when the gender information presented on the document conflicts with its corresponding predefined profile. This controlled annotation process ensures that the labels reflect intentionally introduced inconsistencies rather than subjective judgments based on visual appearance or name-based assumptions.
1. Clarification of generalizability of dataset:
Our manipulation categories are motivated by established identity-document forgery literature, including FantasyID [1] and IDNet [2], which also consider alterations to commonly used identity fields such as names, dates of birth, gender, and identification numbers. Since these fields are shared across identity documents from different countries, the underlying manipulation framework is not inherently restricted to Aadhaar cards. For future work, this provides a foundation for extending FactID to other document templates and national identity systems. Although we compare our forgery type with FantasyID in Appendix A.7. Furthermore, capture-related degradations are excluded from our study, as FactID focuses on digital manipulation performed directly on blank document templates. This allows us to isolate logical inconsistencies without introducing confounding artifacts associated with scanning, printing, or image acquisition.
Accordingly, text–QR discrepancies, acquisition-related degradations, legitimate template variations, and compound manipulations are beyond the current benchmark's scope. In particular, text–QR consistency requires document-specific verification mechanisms that are not uniformly applicable across existing identity-document datasets. We therefore exclude these factors to maintain a controlled evaluation of the selected manipulation categories and comparability with related literature, while recognizing their importance for real-world verification. We acknowledge the reviewer's suggestion and consider extending FactID to encompass these additional scenarios in future work.
2. Task-specific adaptation of vision model:
We agree that conventional image forensic methods are primarily designed to detect visual tampering such as copy-paste, splicing, or image blending. In contrast, FactID intentionally removes these cues by rendering every document directly from a clean template, thereby isolating logic-level inconsistencies as the only source of manipulation. Consequently, as mentioned in Section 4.2, we do not evaluate these methods with the expectation that they will achieve high performance. Rather, the objective is to empirically quantify the limitations of current visual forensic paradigms when confronted with document manipulations that do not leave conventional forensic traces. We believe this evaluation is important because many existing document-based (DocTamper, ASCFormer, ADCDNet) and image forensic methods (TruFor, MMFusion) are implicitly assumed to provide robust protection against manipulated identity documents. Our benchmark demonstrates that this assumption does not hold when the manipulation resides at the semantic or logical level instead of the pixel level, and hence can lead to a “False Sense of Security”. The poor performance of both general image forensic models and document-specific manipulation detectors therefore serves as evidence of a fundamental capability gap, rather than a shortcoming of the benchmark itself. To further adapt the vision models to the specific task of detecting logical manipulation, we fine-tuned DeepID and Effort separately on FactID logical manipulations and tested on FactID as well as traditional forgery-specific ID.
The results highlight the importance of FactID in improving forgery detection beyond conventional visual tampering. Even after fine-tuning on FactID, DeepID and Effort correctly detect only 72.7% and 63.7% of logical forgeries, respectively. Their performance further decreases on LaMa and TELEA visual manipulations, demonstrating limited generalization across forgery types. These findings emphasize that logical and visual manipulations present complementary challenges, highlighting the need for benchmarks such as FactID to develop more comprehensive and robust identity-document verification systems.
[1] Korshunov et al. FantasyID: A dataset for detecting digital manipulations of ID-documents. IEEE IJCB. 2025.
[2] Guan, H., Wang, Y., Xie, L., Nag, S., Goel, R., Erappa Narayana Swamy, N., Yang, Y., Xiao, C., Prisby, J., Maciejewski, R., & Zou, J. (2024). IDNet: A novel dataset for identity document analysis and fraud detection. arXiv preprint arXiv:2408.01690.