acceptodds
Under review as a conference paper at ICLR 2027

Trust the ID, Question the Logic: Rethinking ID Verification

Abstract

Identity documents are the most important line of defense between a verified citizen and an impersonator, and that line is breaking in ways we have not yet learned to see. The global identity verification ecosystem has long anchored its fraud detection capabilities on what can be seen: pixel-level anomalies, morphed photographs, spliced regions, and font irregularities. These are real threats, and the community has built real defenses. But the adversary has moved on. A new class of document fraud, logical manipulation, has quietly outpaced every existing defense. These forgeries carry no visual trace, no textual aberration, and no compression artifact. Instead, they embed contradictions into the semantic fabric of the document itself. They are, by every perceptual measure, perfect fakes. This paper presents the first systematic study of logical manipulation in Aadhaar cards, anticipating emerging threats before they become widespread or exploitable at scale. We introduce , the largest Aadhaar card manipulation dataset, comprising 78,702 samples spanning a rigorously defined taxonomy of template-level and content-level logical manipulations, alongside visual face-level manipulation. Through exhaustive benchmarking of image forensic classifiers, recent manipulation detectors, and state-of-the-art document verification models, we demonstrate that every evaluated system fails, showing that logical manipulation is not a corner case. It is an open frontier, and this work is its first map.

Primary area:
datasets and benchmarks
Keywords:
Logical Manipulation, Identity Fraud, Document Forensics, Fraud Detection, Identity Security

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nthtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces FactID, a dataset of 78,702 generated Aadhaar documents containing template- and content-level inconsistencies. Evaluations of pretrained forensic models, vision-language models, an OCR-based baseline, and human participants highlight limitations in detecting logical errors without conventional visual artifacts. The problem is relevant, but limited coverage and unresolved evaluation issues weaken the contribution.

Critical strengths
  1. The paper addresses a meaningful problem: methods that detect visual artifacts such as splicing, face manipulation, and font irregularities may overlook documents containing errors introduced during generation. Incorporating field validation and cross-field consistency checks into document verification has practical value.

  2. FactID contains 78,702 samples covering template- and content-level errors alongside different face types. Each fake document contains one logical error, enabling clear annotations and targeted analysis of detection failures.

  3. The controlled experiments demonstrate the limited transferability of existing models. When genuine faces are retained and only logical errors are introduced, the 13 forensic models achieve AUCs of 48.78%–52.55%, indicating that these pretrained models cannot directly handle such errors. Separate evaluations of full documents, faces, templates, and content fields provide further diagnostic insight.

Critical weaknesses
  1. FactID contains only Indian Aadhaar cards generated from templates and predefined rules. Without evaluation across countries, document types, or templates, its large sample size does not establish diversity or generalizability.

  2. The benchmark covers only part of document authentication: consistent content does not imply authenticity, and genuine documents may contain clerical errors. It lacks systematic coverage of internally consistent forgeries, text–QR discrepancies, capture degradation, legitimate template variations, and compound or unseen errors.

  3. Most evaluated models target visual tampering and are tested without adaptation, so their failure does not establish the task’s difficulty. The OCR-plus-rules baseline lacks implementation details and error analysis. The concern is not the absence of a new model, but insufficient evidence of the benchmark’s difficulty and research value.

  4. Some results require clarification. ADCD-Net’s aggregate Content Only AUC is 84.32%, while its three category-level AUCs range from 52.17% to 58.69%; this is inconsistent if the same genuine samples and scoring procedure are used. GradCAM alone also cannot establish that 100% AUC reflects “decision collapse.”

  5. Names and facial appearance cannot reliably determine gender, making the “gender mismatch” label questionable. Unadapted-model failures demonstrate limited transfer, while the small human study does not justify claims that logical errors are “invisible to humans.”

Questions for authors
  1. How many templates are represented, and can generalization be tested on another document type or unseen template?
  2. How is gender mismatch defined, and is the required evidence observable in the document?

Besides, please answer the questions listed in weakness.

2 comments

1. Clarification of generalizability of dataset:

Our manipulation categories are motivated by established identity-document forgery literature, including FantasyID [1] and IDNet [2], which also consider alterations to commonly used identity fields such as names, dates of birth, gender, and identification numbers. Since these fields are shared across identity documents from different countries, the underlying manipulation framework is not inherently restricted to Aadhaar cards. For future work, this provides a foundation for extending FactID to other document templates and national identity systems. Although we compare our forgery type with FantasyID in Appendix A.7. Furthermore, capture-related degradations are excluded from our study, as FactID focuses on digital manipulation performed directly on blank document templates. This allows us to isolate logical inconsistencies without introducing confounding artifacts associated with scanning, printing, or image acquisition.

Accordingly, text–QR discrepancies\color{red}{\text{text–QR discrepancies}}, acquisition-related degradations, legitimate template variations, and compound manipulations are beyond the current benchmark's scope. In particular, text–QR consistency requires document-specific verification mechanisms that are not uniformly applicable across existing identity-document datasets. We therefore exclude these factors to maintain a controlled evaluation of the selected manipulation categories and comparability with related literature, while recognizing their importance for real-world verification. We acknowledge the reviewer's suggestion and consider extending FactID to encompass these additional scenarios in future work.

2. Task-specific adaptation of vision model:

We agree that conventional image forensic methods are primarily designed to detect visual tampering such as copy-paste, splicing, or image blending. In contrast, FactID intentionally removes these cues by rendering every document directly from a clean template, thereby isolating logic-level inconsistencies as the only source of manipulation. Consequently, as mentioned in Section 4.2, we do not evaluate these methods with the expectation that they will achieve high performance. Rather, the objective is to empirically quantify the limitations of current visual forensic paradigms\color{blue}{\text{the objective is to empirically quantify the limitations of current visual forensic paradigms}} when confronted with document manipulations that do not leave conventional forensic traces. We believe this evaluation is important because many existing document-based (DocTamper, ASCFormer, ADCDNet) and image forensic methods (TruFor, MMFusion) are implicitly assumed to provide robust protection against manipulated identity documents. Our benchmark demonstrates that this assumption does not hold when the manipulation resides at the semantic or logical level instead of the pixel level, and hence can lead to a “False Sense of Security”. The poor performance of both general image forensic models and document-specific manipulation detectors therefore serves as evidence of a fundamental capability gap, rather than a shortcoming of the benchmark itself. To further adapt the vision models to the specific task of detecting logical manipulation, we fine-tuned DeepID and Effort separately on FactID logical manipulations and tested on FactID as well as traditional forgery-specific ID.

ModelFactID Fake Detected (%)LaMa Fake Detected (%)TELEA Fake Detected (%)
DeepID72.757.463.6
Effort63.760.363.7

The results highlight the importance of FactID in improving forgery detection beyond conventional visual tampering. Even after fine-tuning on FactID, DeepID and Effort correctly detect only 72.7% and 63.7% of logical forgeries, respectively. Their performance further decreases on LaMa and TELEA visual manipulations, demonstrating limited generalization across forgery types. These findings emphasize that logical and visual manipulations present complementary challenges, highlighting the need for benchmarks such as FactID to develop more comprehensive and robust identity-document verification systems.

[1] Korshunov et al. FantasyID: A dataset for detecting digital manipulations of ID-documents. IEEE IJCB. 2025.

[2] Guan, H., Wang, Y., Xie, L., Nag, S., Goel, R., Erappa Narayana Swamy, N., Yang, Y., Xiao, C., Prisby, J., Maciejewski, R., & Zou, J. (2024). IDNet: A novel dataset for identity document analysis and fraud detection. arXiv preprint arXiv:2408.01690.

3. Implementation details of OCR baselines:

We implement a training-free, rule-based verification pipeline using EasyOCR, configured for English and Hindi text recognition. The OCR engine extracts textual information directly from document images, followed by regular-expression-based field extraction and symbolic validation. The evaluation is conducted under three strategies: Full ID, Template Only, and Content Only, using the corresponding test sets. The extracted information is evaluated using three categories of validation rules:

Aadhaar Number Validation: The extracted identifier is checked for its 12-digit structure and mathematical validity using the Verhoeff checksum algorithm. Missing identifiers receive a lower penalty to account for potential OCR extraction failures.

Date Consistency Validation: Extracted dates are parsed using the DD/MM/YYYY format and checked against chronological constraints, including whether the date of birth precedes the issue date, whether the issue date is not in the future, and whether the calculated age falls within a plausible range (0–120 years).

Template Validation: Fixed institutional information, including authority names, official website information, and predefined spelling variations, is verified using string matching and fuzzy similarity through Python's SequenceMatcher, with a similarity threshold of 0.8 for principal template strings.

Each detected violation contributes a predefined penalty, and the accumulated violation score is compared against a fixed threshold of 0.4 to classify the document as genuine or manipulated. The continuous violation score is used to compute AUC.

4. Justification of 100% AUC and gender mismatch:

We agree that 100% AUC and 0% EER indicate perfect score-based discrimination on the evaluated samples, and we clarify the interpretation in Appendix A.5–A.6. However, perfect discrimination does not necessarily imply that a model identifies the underlying logical manipulation, as predictions may rely on incidental visual cues or spurious correlations. This distinction is also illustrated in Figure 1,where VLMs correctly classify forged documents while providing incorrect or irrelevant justifications.\color{blue}{\text{where VLMs correctly classify forged documents while providing incorrect or irrelevant justifications.}} Such cases demonstrate that correct classification does not necessarily reflect an understanding of genuine forgery, leading to "decision collapse".

Regarding gender mismatch, the ground-truth labels are manually annotated based on the predefined demographic profiles established during dataset construction\color{blue}{\text{the ground-truth labels are manually annotated based on the predefined demographic profiles established during dataset construction}}, rather than inferred solely from facial appearance or names. Each identity document is associated with an assigned demographic profile, which serves as the reference for evaluating gender consistency. A gender mismatch is introduced when the gender information presented on the document conflicts with its corresponding predefined profile. This controlled annotation process ensures that the labels reflect intentionally introduced inconsistencies rather than subjective judgments based on visual appearance or name-based assumptions.

Reviewer_1s5aOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces FactID, a benchmark for evaluating logical manipulation in Aadhaar identity documents. The dataset contains 78,702 samples covering template-level inconsistencies, content-level contradictions, and facial manipulations, including morphed and synthetic faces. The authors evaluate a range of pretrained image forensic detectors, document-specific manipulation detectors, vision-language models, and an OCR-based rule-driven baseline. The results reveal substantial limitations in existing models when detecting logically inconsistent identity documents, particularly when conventional visual manipulation cues are absent.

Critical strengths
  1. Meaningful problem motivation. The paper highlights an important limitation of conventional identity document verification: visually plausible documents may contain logical inconsistencies that cannot be reliably identified through image-level forensic analysis alone.

  2. Large-scale dataset with diverse manipulation categories. FactID provides 78,702 samples covering different facial conditions and several template-level and content-level manipulations. The proposed taxonomy offers a useful starting point for studying logical consistency in identity documents.

  3. Broad empirical evaluation. The paper evaluates multiple image forensic and document-trained models under four input strategies. The per-manipulation results and OCR-based comparison provide useful observations about the limitations of existing visual detectors.

Critical weaknesses
  1. Limited realism and generalizability of the dataset. Although the paper refers to 2,326 samples as genuine Aadhaar cards, Section 3.1 and Appendix A.2 indicate that all documents are synthetically constructed from a blank template with assigned demographic information and externally sourced face images. Consequently, the benchmark does not capture the full variability of genuinely issued and physically acquired identity documents, raising concerns about its representativeness and generalization to real-world verification scenarios.

  2. Ambiguity in the definition of logical manipulation. Several manipulation types, including spelling errors, word jumbling, and 13-digit Aadhaar numbers, involve directly observable textual or formatting violations rather than complex semantic inconsistencies. Moreover, the validity of gender-mismatch labels requires careful justification, since gender cannot be reliably inferred from facial appearance or names alone. These issues weaken the claim that all manipulations are visually indistinguishable and objectively verifiable.

  3. Insufficiently representative baselines for logical verification. Most evaluated models are pretrained for visual forgery detection and are tested without task-specific adaptation, although they are not explicitly designed to verify document-level logical constraints. The OCR-based baseline is a useful addition, but its relatively poor performance on content-level manipulations is not sufficiently analyzed. Stronger OCR-based validation systems and document-reasoning baselines would be necessary to establish the difficulty of the proposed task more convincingly.

  4. Potential confounding factors in the evaluation protocol. Under the Face Only strategy, documents containing genuine, unmodified faces but logically manipulated fields remain labeled as fake, although the manipulation is not observable in the provided input. Furthermore, the paper describes a random 70:30 train-test split without clearly establishing whether identities, source faces, or generated variants are separated across splits. These issues may affect the interpretability and reliability of the reported results.

  5. Questionable interpretation of certain experimental results. Tables 4 and 6 report AUC values of 100% for several models on word-jumbling manipulations, accompanied by EER values of 0% in the appendix. However, Appendix A.5–A.6 interprets these results as prediction collapse. Perfect AUC and zero EER indicate perfect score-based discrimination on the evaluated samples, although the discrimination may rely on spurious features. The GradCAM visualizations alone do not establish the underlying cause. This interpretation requires clarification and more rigorous analysis.

  6. Limited validation of human performance and statistical reliability. The human perceptual study involves only 34 participants, each evaluating seven documents, yet the paper draws broad conclusions about human inability to detect logical manipulation. The evaluation lacks confidence intervals and detailed analysis of participant variability. Similarly, the main model comparisons do not report uncertainty estimates, making it difficult to assess the reliability of the observed performance differences.

Questions for authors
  1. How well does FactID represent real-world identity verification scenarios, given that both genuine and manipulated documents are synthetically constructed? Have the authors considered evaluating on physically captured documents or additional template variants?

  2. How are gender-mismatch labels objectively established, and how do the authors distinguish visually observable formatting violations from genuinely knowledge-intensive logical contradictions?

  3. Could the authors provide stronger OCR-based or document-reasoning baselines, particularly for deterministic checks such as date consistency and identifier validation? How would these systems compare with the evaluated forensic detectors?

  4. Could the authors clarify the interpretation of the 100% AUC and 0% EER results, explain the source-level train-test splitting protocol, and provide statistical uncertainty estimates for the main experiments?

3 comments

1. Realism and Generalizability of FactID:

First of all, Aadhaar cards contain highly sensitive personal information, so collecting and publicly distributing genuine, physically issued identity documents raises significant privacy, ethical, and legal concerns. Identity document logical manipulation can be done through two mechanisms: (i) Digital Manipulation, where an existing digital ID or its template is directly modified to introduce logically inconsistent information, and (ii) Physical Manipulation, where a printed ID is scanned, its information is altered logically, and the manipulated document is subsequently reprinted. FactID specifically focuses on the first scenario, where logical inconsistencies are introduced during digital document generation.\color{blue}{\text{FactID specifically focuses on the first scenario, where logical inconsistencies are introduced during digital document generation}}. As described in Appendix A.2, we construct clean Aadhaar cards using independently assigned demographic information without incorporating any individual's real personal data. Importantly, FactID is not generated by merely duplicating the 2,326 clean samples. Instead, each forged document is constructed through different combinations of facial conditions (real, morphed, or synthetic), logical manipulation types, and document attributes, ensuring systematic coverage of representative forgery scenarios.

Physical manipulation, in contrast, may introduce additional visual artifacts arising from scanning, printing, compression, blurring, and other acquisition-related degradations. These artifacts can act as confounding factors, potentially allowing forensic models to detect the document through visual traces rather than identifying the underlying logical inconsistency\color{blue}{\text{potentially allowing forensic models to detect the document through visual traces rather than identifying the underlying logical inconsistency}}. Therefore, FactID deliberately focuses on digital generation-time manipulation to isolate logical inconsistencies from conventional visual forgery artifacts, enabling a more controlled and interpretable evaluation of logical forgery detection.

Additionally, our manipulation categories are motivated by established identity-document forgery literature, including FantasyID [1] and IDNet [2], which also consider alterations to commonly used identity fields such as names, dates of birth, gender, and identification numbers. Since these fields are shared across identity documents from different countries, the underlying manipulation framework is not inherently restricted to Aadhaar cards. This provides a foundation for extending FactID to other document templates and national identity systems in future work.

2. Clarification of the definition of logical manipulation:

The selected logical manipulations probe different aspects of document validity, including incorrect template information, invalid field values, and inconsistencies between document elements. The central motivation is that visual plausibility does not guarantee logical validity, particularly as generative technologies make convincing document imagery increasingly accessible. This concern is also recognized by CogniCred [3], which motivates cognitive credential-forgery detection beyond visual tampering cues. We acknowledge that some manipulations are textually observable; however, identifying them requires knowledge of document-specific validation rules. At the same time, these minute issues might go unnoticed unless users are aware of these issues and look at every possible detail in the card at a minute level.\color{blue}{\text{At the same time, these minute issues might go unnoticed unless users are aware of these issues and look at every possible detail in the card at a minute level.}}

For example, a 13-digit Aadhaar number may appear plausible to an unfamiliar observer\color{blue}{\text{a 13-digit Aadhaar number may appear plausible to an unfamiliar observer}}. Importantly, identifier formats can evolve, as demonstrated by changes in India’s Voter ID (EPIC) numbering system and Vietnam’s transition from 9-digit to 12-digit IDs\color{red}{\text{as demonstrated by changes in India's Voter ID (EPIC) numbering system and Vietnam's transition from 9-digit to 12-digit IDs}}, while the old 9-digit ID was still valid until it expired. These examples motivate proactive evaluation of such inconsistencies rather than assuming universal knowledge of fixed rules. Similarly, spelling errors and word jumbling represent plausible errors in AI-generated documents, motivating their inclusion. We distinguish these template-level violations from content-level logical contradictions. Regarding gender mismatch, labels are manually annotated against the predefined demographic profiles during dataset construction, rather than inferred solely from facial appearance or names.

[1] Korshunov et al. FantasyID: A dataset for detecting digital manipulations of ID-documents. IEEE IJCB. 2025.

[2] Guan, H., Wang, Y., Xie, L., Nag, S., Goel, R., Erappa Narayana Swamy, N., Yang, Y., Xiao, C., Prisby, J., Maciejewski, R., & Zou, J. (2024). IDNet: A novel dataset for identity document analysis and fraud detection. arXiv preprint arXiv:2408.01690.

[3] Li, J., Sun, J., Chen, W., Wang, Z., Zhou, S., Bu, J., ... & Wang, W. (2026, September). CogniCred: A Dataset and Benchmark for Cognitive Credential Forgery Detection. In European Conference on Computer Vision (pp. 624-640). Cham: Springer Nature Switzerland.

3. Distinguishing visually observable formatting violations from genuinely knowledge-intensive logical contradictions:

To further distinguish logical manipulation from visually observable tampering, we additionally created a baseline by manipulating real IDs from FactID using conventional editing operations such as text inpainting. We then evaluated representative vision forensic models and VLMs on both edited IDs and our logically manipulated IDs.

ModelFake Accuracy on Edited IDs (%)Fake Accuracy on FactID (%)
Effort406
TruFor60
Llama-3.2-Vision336

The significant reduction in fake detection accuracy from edited IDs to FactID indicates that these models primarily rely on visible editing artifacts, whereas our benchmark removes such cues and requires reasoning over logical inconsistencies for forgery detection. Logical manipulations such as invalid DOB, incorrect Aadhaar spelling, and invalid details date are detected much more confidently by vision models and LLMs when they are introduced through editing existing IDs but are largely missed in FactID where no visual editing artifacts are present.

4. Task-specific adaptation of vision model:

We agree that conventional image forensic methods are primarily designed to detect visual tampering such as copy-paste, splicing, or image blending. In contrast, FactID intentionally removes these cues by rendering every document directly from a clean template, thereby isolating logic-level inconsistencies as the only source of manipulation. Consequently, as mentioned in Section 4.2, we do not evaluate these methods with the expectation that they will achieve high performance. Rather, the objective is to empirically quantify the limitations of current visual forensic paradigms\color{blue}{\text{the objective is to empirically quantify the limitations of current visual forensic paradigms}} when confronted with document manipulations that do not leave conventional forensic traces. We believe this evaluation is important because many existing document-based (DocTamper, ASCFormer, ADCDNet) and image forensic methods (TruFor, MMFusion) are implicitly assumed to provide robust protection against manipulated identity documents. Our benchmark demonstrates that this assumption does not hold when the manipulation resides at the semantic or logical level instead of the pixel level, and hence can lead to a “False Sense of Security”. The poor performance of both general image forensic models and document-specific manipulation detectors therefore serves as evidence of a fundamental capability gap, rather than a shortcoming of the benchmark itself. To further adapt the vision models to the specific task of detecting logical manipulation, we fine-tuned DeepID and Effort separately on FactID logical manipulations and tested on FactID as well as traditional forgery-specific ID.

ModelFactID Fake Detected (%)LaMa Fake Detected (%)TELEA Fake Detected (%)
DeepID72.757.463.6
Effort63.760.363.7

The results highlight the importance of FactID in improving forgery detection beyond conventional visual tampering. Even after fine-tuning on FactID, DeepID and Effort correctly detect only 72.7% and 63.7% of logical forgeries, respectively. Their performance further decreases on LaMa and TELEA visual manipulations, demonstrating limited generalization across forgery types. These findings emphasize that logical and visual manipulations present complementary challenges, highlighting the need for benchmarks such as FactID to develop more comprehensive and robust identity-document verification systems.

5. Misinterpretation of Face-only strategy:

The reviewer’s interpretation of the Face-only strategy does not reflect our evaluation protocol\color{red}{\text{The reviewer’s interpretation of the Face-only strategy does not reflect our evaluation protocol}}. As explicitly stated in Section 4.1, a single, fixed set of clean reference samples is shared across all four strategies, while the fake samples are selected according to the manipulation type. Consequently, documents containing unmodified faces but logically manipulated fields are excluded from the Face Only evaluation and are never incorrectly labeled as facial forgeries.

Regarding data partitioning, we first construct the four strategy-specific datasets (Full ID, Face Only, Template Only, and Content Only) and subsequently apply a separate 70:30 train–test split within each strategy. Our protocol is designed for strategy-wise evaluation of distinct manipulation types, rather than identity- or source-disjoint generalization. We believe these clarifications address the concerns regarding the Face Only labels and the interpretation of the evaluation protocol.

6. Justification of 100% AUC:

We agree that 100% AUC and 0% EER indicate perfect score-based discrimination on the evaluated samples, and we clarify the interpretation in Appendix A.5–A.6. However, perfect discrimination does not necessarily imply that a model identifies the underlying logical manipulation, as predictions may rely on incidental visual cues or spurious correlations. This distinction is also illustrated in Figure 1, where VLMs correctly classify forged documents while providing incorrect or irrelevant justifications. Such cases demonstrate that correct classification does not necessarily reflect genuine forgery understanding. As an example, when an ID card with a spelling error ("Government" to "Govenrment") is evaluated through Qwen2.5-VL-7B-Instruct, it flags the ID as fake, but its explanation is "the background pattern is inconsistent, suggesting it might be a template or a mock-up rather than an official document". However, it fails to identify the actual spelling manipulation responsible for the forgery. This example highlights that correct binary classification does not necessarily imply correct identification or understanding of the underlying manipulation. Similarly, our GradCAM visualizations suggest that forensic models may focus on regions unrelated to the actual forgery. Thus, despite producing correct classification scores, models may rely on irrelevant visual cues rather than the intended logical inconsistencies. We refer to this discrepancy between classification outcomes and forgery-relevant evidence as a failure of forgery localization and interpretation, rather than prediction collapse in the strict classification sense.

7. Uncertainty estimation of main model comparison & human evaluation:

ModelTemplate Only AUC (%) [95% CI] ↑Content Only AUC (%) [95% CI] ↑
DeepID41.43 [39.51, 43.35]19.11 [18.19, 20.03]
Effort49.90 [47.70, 50.74]66.40 [65.70, 68.31]
ADCD-Net49.87 [47.84, 51.95]84.32 [82.21, 85.26]
ASCFormer19.63 [18.75, 20.50]73.71 [72.75, 74.67]
---------
Human evaluation45.10 [39.0. 51.6] (Logical manipulation)

The results reveal substantial variability in forensic model performance across template-level and content-level logical manipulations. Under the Template Only strategy, DeepID (41.43%), Effort (49.90%), ASCFormer (19.63%) and ADCD-Net (49.87%) exhibit below-chance AUC values. The corresponding 95% confidence intervals further support these observations, with only ADCD-Net's and Effort's intervals containing 50%.

For the Content Only strategy, performance varies considerably, ranging from 19.11% for DeepID to 84.32% for ADCD-Net. Although Effort (66.40%), ADCD-Net (84.32%), and ASCFormer (73.71%) achieve higher AUC values, these aggregate scores do not necessarily establish that the models correctly identify the underlying logical inconsistencies, as their predictions may also be influenced by visual or preprocessing-related artifacts.

Furthermore, the human evaluation achieves an AUC of 45.10% (95% CI: [39.0%, 51.6%]), indicating near-chance discrimination of logically manipulated documents. Since the confidence interval includes 50%, the results do not establish a statistically significant difference from chance-level performance. These findings suggest that logical inconsistencies can be challenging for human evaluators as well as automated forensic models.

Overall, these results demonstrate inconsistent model behavior across logical manipulation strategies, highlighting the importance of FactID as a benchmark for systematically evaluating the robustness and limitations of existing document forgery detectors.

Reviewer_HZ8TOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces FactID, a synthetic Aadhaar ID dataset that combines visual forgeries, such as deepfakes and inpainting, with logical inconsistencies. It distinguishes template-level manipulations, including spelling errors and altered labels, from content-level manipulations, such as impossible dates, invalid ID numbers, and gender mismatches. The authors evaluate image-forgery detectors, tampering-localization methods, document-specific models, and a rule-based OCR baseline using full IDs and selectively masked inputs. Results show that existing methods detect some logical manipulations inconsistently, with no tested approach reliably covering all forgery types.

Critical strengths

None

Critical weaknesses

Overall, I think this paper should clearly be rejected because it fails to establish a strong motivation for the chosen logical manipulation, evaluates methods poorly matched to the benchmarked task, contains contradictory claims, contains claims unsupported by the results due to poor experimental design (lack of ablations), does not include key details regarding the experimental procedures.

Details:

  • The paper does not justify its chosen logical manipulations; several appear detectable through simple template checks or checksum validation.
  • Most evaluated methods detect visual tampering rather than logical inconsistencies, making their failure unsurprising and insufficiently informative.
  • Claims of visually indistinguishable forgeries conflict with explanations based on visual artifacts. Missing ablations prevent separating logical-error detection from artifact detection.
  • The OCR baseline, human study, and VLM evaluations omit procedures needed to assess their validity and reproducibility. -Face-image licensing, consent, misuse safeguards, and conflicting dataset release terms are insufficiently addressed.
Questions for authors
  • What motivated the choice of methods evaluated in this benchmark?
  • What motivated the selection of logical manipulations included in the benchmark?
  • Could you describe the OCR baseline procedure, including the OCR engine, input regions, and validation rules?
  • Could you include an ablation that separates detection of rendering artifacts introduced by the ID generation pipeline from detection of logical errors? In particular, the replacement operations used for template-level manipulations may introduce visual artifacts that trigger the evaluated detectors under the proposed input strategies.
  • How will you ensure that dataset distribution complies with KDEF licensing requirements, respects the copyright of images in UTKFace, and mitigates misuse risks?
  • Could you clarify the statement in Appendix A.6 that an AUC of 100% does not indicate success? This appears to indicate perfect separation on the benchmark task, although it does not establish whether the model detects logical errors or incidental artifacts. More generally, several reported AUC values are substantially above or below 50%, suggesting that the models distinguish authentic and manipulated samples, potentially with reversed score direction.
4 comments
AuthorsOpenReview ↗

Official comments from authors

1. Central motivation of choosing logical manipulation:

The central motivation is that visual plausibility does not guarantee logical validity, particularly as generative technologies make convincing document imagery increasingly accessible. This concern is also recognized by CogniCred [1], which motivates cognitive credential-forgery detection beyond conventional visual tampering cues. Although certain violations, such as invalid identifier lengths or checksum errors, may appear straightforward to detect through predefined rules, such validation presupposes knowledge of the document's country-specific specifications and verification requirements. It is not reasonable to assume that every individual, particularly non-experts or people unfamiliar with India's identification system, knows the structural and logical requirements of an Aadhaar card. For instance, knowing that an Aadhaar number contains exactly 12 digits and cannot begin with 0 or 1 requires specific domain knowledge. The existence of deterministic validation rules does not make the broader problem trivial; rather, it highlights the need for accessible and reliable document-verification mechanisms. Our contribution is therefore to systematically expose and characterize the distinction between visually convincing documents and logically valid content through controlled rendering. Although FactID focuses on Indian Aadhaar cards, the underlying problem is not country-specific: government-issued identity documents worldwide have their own structural, semantic, and consistency requirements. Thus, the verification challenges studied in FactID are relevant to the broader problem of protecting government-issued credentials against increasingly convincing AI-generated forgeries.

2. Rationale behind use of the image forensics and document-specific models:

We agree that conventional image forensic methods are primarily designed to detect visual tampering introduced when an existing document is manipulated through operations such as copy-paste, text overwriting, splicing, or image blending. In contrast, FactID intentionally removes these cues by rendering every document directly from a clean template, thereby isolating logic-level inconsistencies as the only source of manipulation. Consequently, as mentioned in Section 4.2, we do not evaluate these methods with the expectation that they will achieve high performance. Rather, the objective is to empirically quantify the limitations of current visual forensic paradigms when confronted with document manipulations that do not leave conventional forensic traces. We believe this evaluation is important because many existing document-based (DocTamper, ASCFormer, ADCDNet) and image forensic methods (TruFor, MMFusion) are implicitly assumed to provide robust protection against manipulated identity documents. Our benchmark demonstrates that this assumption does not hold when the manipulation resides at the semantic or logical level instead of the pixel level, and hence can lead to a “False Sense of Security”. The poor performance of both general image forensic models and document-specific manipulation detectors therefore serves as evidence of a fundamental capability gap, rather than a shortcoming of the benchmark itself.

3. Ablation: Separating logical manipulation from visible artifacts:

To separate logical manipulation detection from artifact detection, we additionally created a baseline by manipulating existing IDs using conventional editing operations such as text inpainting. We then evaluated representative vision forensic models and VLMs on both edited IDs and our logically manipulated IDs.

ModelFake Accuracy on Edited IDs (%)Fake Accuracy on FactID (%)
Effort406
TruFor60
Llama-3.2-Vision336

The significant reduction in fake detection accuracy from edited IDs to FactID indicates that these models primarily rely on visible editing artifacts, whereas our benchmark removes such cues and requires reasoning over logical inconsistencies for forgery detection. Logical manipulations such as invalid DOB, incorrect Aadhaar spelling, and invalid details date are detected much more confidently by vision models and LLMs when they are introduced through editing existing IDs, but are largely missed in FactID, where no visual editing artifacts are present.

[1] Li, J., Sun, J., Chen, W., Wang, Z., Zhou, S., Bu, J., ... & Wang, W. (2026, September). CogniCred: A Dataset and Benchmark for Cognitive Credential Forgery Detection. In European Conference on Computer Vision (pp. 624-640). Cham: Springer Nature Switzerland.

AuthorsOpenReview ↗

Official comments from authors

4. Details of OCR baseline, human, and VLM model evaluation:

(a) OCR baseline evaluation: We implement a training-free, rule-based verification pipeline using EasyOCR, configured for English and Hindi text recognition. The OCR engine extracts textual information directly from document images, followed by regular-expression-based field extraction and symbolic validation. The evaluation is conducted under three strategies: Full ID, Template Only, and Content Only, using the corresponding test sets. The extracted information is evaluated using three categories of validation rules:

Aadhaar Number Validation: The extracted identifier is checked for its 12-digit structure and mathematical validity using the Verhoeff checksum algorithm. Missing identifiers receive a lower penalty to account for potential OCR extraction failures.

Date Consistency Validation: Extracted dates are parsed using the DD/MM/YYYY format and checked against chronological constraints, including whether the date of birth precedes the issue date, whether the issue date is not in the future, and whether the calculated age falls within a plausible range (0–120 years).

Template Validation: Fixed institutional information, including authority names, official website information, and predefined spelling variations, is verified using string matching and fuzzy similarity through Python's SequenceMatcher, with a similarity threshold of 0.8 for principal template strings.

Each detected violation contributes a predefined penalty, and the accumulated violation score is compared against a fixed threshold of 0.4 to classify the document as genuine or manipulated. The continuous violation score is used to compute AUC.

(b) Human perceptual study: As already described in the Introduction (Section 1, Humans Are Not the Fallback), we conduct a controlled perceptual study involving 34 participants. Each participant is presented with seven different identity documents randomly sampled from FactID and is explicitly informed beforehand that the documents may contain template-level inconsistencies, content-level logical inconsistencies, and/or face forgery. Participants are then asked to determine whether each document is genuine or manipulated based solely on visual inspection, without automated verification assistance. We analyze participants' binary predictions against the ground-truth labels. The reported human AUC is 45.10%, reflecting the limited discrimination observed in this controlled evaluation. The aggregated results are summarized below.

EvaluationAUC ↑Accuracy ↑APCER ↓BPCER ↓
Human Participants45.10%57.98%32.35%77.45%

A closer inspection of the error rates reveals an important observation. The participants adopted a conservative decision strategy, frequently suspecting genuine documents to be forged rather than reliably identifying the actual logical inconsistencies.

(c) VLM evaluation: We evaluate two multimodal vision-language models, Qwen2.5-VL-7B-Instruct and Llama-3.2-11B-Vision-Instruct, for document authenticity assessment. Both models are provided with identity document images and instructed to evaluate authenticity based on logical consistency and visual evidence. As illustrated in Figure 1 of the manuscript, the evaluation uses the following standardized prompt:

Is this ID card fake or real? Answer with: Prediction: [Fake/Real] Justification: [One sentence]. Focus on logical consistency and visual artifacts.

This prompt requires the models to provide both a binary authenticity prediction and a brief justification, allowing us to examine not only whether the model correctly classifies a document but also whether its explanation identifies the underlying logical inconsistency. As shown in Figure 1, VLMs may correctly classify a forged document while providing incorrect justifications. This also addresses the reviewer's concern regarding 100% AUC. While 100% AUC indicates perfect discrimination, it does not guarantee that the model identifies the actual forgery. Models may rely on spurious visual cues rather than logical inconsistencies. Thus, correct predictions do not necessarily imply genuine forgery understanding.

5. Dataset Licensing, Privacy, and Misuse Prevention:

We use KDEF and UTKFace solely for non-commercial scientific research, consistent with their usage conditions. KDEF permits researchers flexibility beyond these restrictions for academic research. Our use of these datasets is strictly limited to academic research, with no commercial intent. FactID will be released subject to applicable copyright and redistribution permissions, under a restricted-use agreement prohibiting misuse. We will consider visible watermarks to prevent fraudulent use and provide a mechanism for individuals to request removal of their facial images due to privacy concerns.

AuthorsOpenReview ↗

Licensing, Privacy, fair Usage

We want to highlight that datasets captured across multiple public platforms, such as YouTube (YouTube-VOS, YouTube-VIS, and TalkingHead-1KH), are still widely used in computer vision research. Since the videos are sourced from a public website, we adhere to the Directives, including European Directive 2019/7901, which allows researchers to use them. We verified that source channels do not explicitly forbid AI training. Since the videos are obtained from a public website, we comply with the European Directive 2019/7901, which permits researchers to use and store data from the public web domain for non-commercial research purposes. Moreover, dataset owners do not restrict the academic use of their datasets.

Further, several other measures will be taken to prevent misuse; the first is, obviously, the license agreement. The images might be watermarked as needed without compromising the intended task.

Reviewer_HZ8TOpenReview ↗

Feedback to comment

I want to thank the authors for writing a thorough response to my comments. FactID is presented mainly as a benchmark for machine verification systems. However, the ease of detection by non-expert individuals is not a good indicator of the difficulty for a fraud-detection algorithm, because complex algorithmic tasks such as checksum verification can be very difficult for a person to do at a glance while remaining trivial for existing computer systems. I believe that most, if not all, tasks in FactID belong to this category. I am also not convinced that it can be reasonably assumed that those methods can detect logical manipulations. Additionally, a value of 0% accuracy for TruFor indicates that the model can almost perfectly separate samples with flipped predictions. Since TruFor shows this separation in the edits baseline, this indicates that the edits introduce cues that allow discrimination between classes. This also matches explanations of AUC proposed in the authors’ responses to reviewers ntht and 1s5a. Therefore, the models under test seem to able to solve this benchmark by ignoring the logic manipulations and relying on those cues instead. Neither the paper nor any of the responses propose a quantitative way to separate this shortcut from what the benchmark claims to test.

Looking at the FAQ for KDEF, the dataset can be used for non-commercial research without any additional restriction beyond "non-commercial", but it may not be redistributed without permission from the authors. Since the KDEF license explicitly prohibits redistribution without permission, I do not see how the restrictive license on FactID can make its distribution compliant with that restriction. A solution could be to publish source code for FactID generation instead of data. Another more common practice (for example, in Conceptual Captions SBU, LAION, YFCC) is to distribute URLs to download the contents.The proposed watermarking of the ID images and an opt-out mechanism could indeed reduce the risk of dataset misuse.

Because of these issues, I have decided to keep my score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Discussion and Acknowledgement

We sincerely thank the reviewers for providing constructive comments even with these hectic deadline schedules.

But as we are approaching towards the end of this critical discussion period, it would be great if you can revisit our response and provide us positive acknowledgement or provide us the timely opportunity to resolve any new/remaining comments.

We have thoroughly addressed each concern raised by all the reviewers.

Dear reviewers,

As the discussion period approached the end, please take a moment to read the authors’ responses and share your thoughts on any remaining points of discussion. If the discussion has changed your assessment, please update your review accordingly.

Thank you for your time and contributions!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.