acceptodds
Withdrawn from ICLR 2027

ForgePan-5.9M: a Generalizable Benchmark for Image Forgery Detection and Localization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Rapid advancements in AI-Generated Content (AIGC) are blurring the boundaries of reality and the synthetic. AIGC can be a fully generated image, partially manipulated by either AI or physical editing tools, or a mixture of both, which we term “hybrid edits”. Despite the increasing volume of AIGC detection benchmarks, prior efforts struggle to generalize to the granularity of manipulation types, such as detectors being reliable for fully generated images may perform poorly on partially edited images, or vice versa, nor is there a unified detection model that can generalize well across different real-world manipulations in terms of both detection and localization. Existing methods generalize poorly to hybrid edits which are multiturn edited images with both physical edit and AI edit. We first introduce ForgePan-5.9M, a large-scale detection and localization benchmark featuring data-aligned samples across multiple manipulation types, such as AI editing, physical copy/move, and full-image generation, enabling unified evaluation of both manipulation detection and localization. This aligned dataset aided with proposed ForgeFormer that learns pixel-level intrinsic fingerprints associated with diffusion, GANs, or physically edited forgeries and ForgeFormer simultaneously achieves state-of-the-art in identifying and localizing unseen fully generated AI images and partial manipulations by either AI or physical edits and generalize well on hybrid edits. Notably, ForgeFormer achieves 1st rank with 87% balanced accuracy on the IEDAL2 challenge dataset, further validating its effectiveness in realistic, heterogeneous forgery scenarios.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Deepfake detection and Localization, Multi-turn manipulation localization, Cross-dataset generalization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rCqROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper targets the limitations of traditional binary image forgery localization, which only distinguishes real and fake pixels but fails to identify diverse forgery types and locate mixed tampered regions in a single image. The authors make two core contributions: a large-scale benchmark dataset ForgePan-5.9M with 5.975 million annotated images covering natural scenes, physical editing, multiple AI editing modes, and hybrid tampering, and a panoramic segmentation-based ForgeFormer model built on DINOv3. The proposed method unifies image-level forgery type classification and pixel-level tampering source localization, and adopts multi-scale feature fusion, mask projection, supervised contrastive learning and deep clustering to optimize fine-grained forgery reasoning. Extensive experiments on in-domain test sets, cross-generator benchmarks, and the IEDAL2 hidden test verify the effectiveness and generalization ability of the method for complex hybrid image forgery detection.

Critical strengths
  • Novel and practical task formulation: Different from conventional binary forgery segmentation, this work redefines image forgery detection as a multi-source panoramic segmentation task. It enables the model to distinguish physical editing, different AI editing styles and hybrid tampering regions simultaneously, which better fits real-world complex forgery scenarios.

  • Systematic dataset construction: The ForgePan-5.9M dataset adopts a well-designed aligned editing strategy, applying various tampering operations on identical mask regions. This design mitigates the confounding bias between content features and forgery traces, providing a high-quality and balanced benchmark for fine-grained image forensic research.

  • Task-aligned model architecture: The ForgeFormer model integrates DINOv3 pre-trained features, multi-scale feature pyramids, iterative mask projection, and auxiliary contrastive learning and clustering constraints. Ablation experiments fully validate the positive contribution of each core module to multi-source forgery localization.

  • Comprehensive experimental evaluation: The paper conducts extensive evaluations including in-domain quantitative results, cross-generator generalization tests, public benchmark comparisons, hidden blind tests on IEDAL2, ablation studies, and robustness tests against common image post-processing, demonstrating solid overall performance.

Critical weaknesses
  • Unverified causal effectiveness of aligned data construction: The core claim that mask-aligned editing improves generalization lacks strict controlled experiments. The current dataset cannot exclude interfering factors such as inconsistent generator characteristics, boundary fusion strategies and compression differences across editing branches, so the genuine promotion mechanism of the aligned design is unproven.

  • Limited generalization from synthetic to real-world scenarios: Most training and test data are synthetically generated under fixed pipelines. The obvious performance drop on the IEDAL2 hidden test (pixel-level F1 drops to 60.3%) reveals a significant gap between synthetic data performance and real complex tampering scenarios with multi-round editing and arbitrary post-processing.

  • Inadequate interpretability of auxiliary modules: The online deep clustering module brings performance gains, but the paper fails to verify whether the clustered features correspond to intrinsic forgery fingerprints of different generators. The actual working mechanism of this module remains ambiguous.

  • Asymmetric baseline comparison and incomplete evaluation: Traditional binary segmentation baselines lack the multi-source classification output space, leading to unfair performance comparison. Besides, the model’s fine-grained errors (e.g., misclassification of tampering sources, boundary deviation) and performance under complex combined perturbations are not fully analyzed.

  • High training and deployment costs: The model relies on a large DINOv3 backbone and multiple auxiliary training objectives, requiring high-end GPU resources for long-duration training. The paper lacks efficiency analysis of model parameters, inference latency and lightweight deployment potential, limiting its practical application value.

Questions for authors

Please refer to the weaknesses.

Reviewer_BnDnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ForgePan-5.9M, a large-scale benchmark covering various image manipulations, including AI-generated, physically edited, and hybrid images. It also proposes ForgeFormer, a DINOv3-based panoptic segmentation framework that combines mask projection, deep clustering, and contrastive learning for unified forgery detection and localization. Experiments on public and out-of-distribution benchmarks show substantial improvements over existing methods, supported by ablation studies.

Critical strengths
  1. ForgePan-5.9M covers a wide range of manipulation types, including hybrid edits, which are often overlooked by existing benchmarks. Its scale and unified annotations make it a potentially valuable resource for the community.

  2. ForgeFormer achieves substantial improvements over existing methods across multiple benchmarks, particularly on unseen manipulations and hybrid edits. The extensive evaluations and ablation studies provide encouraging evidence of its effectiveness.

Critical weaknesses

1. Limited methodological novelty. ForgeFormer mainly combines existing components, including DINOv3, Mask2Former, contrastive learning, and deep clustering. While the integration is effective, the paper offers limited new methodological insights beyond adapting these techniques to forgery localization.

2. Potential dataset biases. Despite the proposed data alignment strategy, the dataset still contains correlations between manipulation classes, source datasets, generators, and post-processing operations (Appendix J). These shortcuts may allow the model to learn dataset-specific artifacts rather than generalizable forgery features.

3. Insufficient evidence for the benefits of data alignment. A key motivation is that operator-aligned data improves generalization. However, the paper lacks a controlled comparison between aligned and unaligned training data. It is therefore difficult to tell how much of the improvement actually comes from the proposed dataset design.

4. Concerns about baseline comparisons. Many existing methods are designed for binary forgery localization rather than multi-class panoptic segmentation. Although the authors adapt these methods and provide additional ablations, the main comparisons still favor ForgeFormer's output formulation. This makes it difficult to fully separate better forensic representations from the advantages of the task formulation itself.

Minor: First, the hybrid edits are primarily constructed using spatially separate manipulation regions, which may not fully reflect more complicated real-world editing workflows. Second, the robustness evaluation only considers a single strength for each post-processing operation. Third, the failure analysis shows that detecting very small manipulated regions remains challenging. Fourth, the paper would benefit from clearer figures and a more concise explanation of the architecture. Finally, reporting variance across multiple training runs would help establish the reliability of the experimental improvements.

Questions for authors
  1. Can the authors provide a controlled comparison between aligned and unaligned training data, keeping dataset size, source composition, and generators fixed? This would help establish whether data alignment actually contributes to the reported generalization improvements.

  2. Appendix J acknowledges several potential shortcuts related to source datasets, generators, and post-processing. Can the authors provide evidence that ForgeFormer learns genuine manipulation fingerprints rather than relying on these dataset-specific cues?

  3. How does ForgeFormer compare with a standard DINOv3 + Mask2Former baseline trained with the same data, supervision, and computational budget? This comparison would help clarify how much improvement comes from the proposed architectural components.

  4. In Table 3, removing the image-level classification loss still gives a macro-F1 of 92.60%. Since this loss is used to train the classification head, how is this result obtained?

Reviewer_5iHAOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ForgePan-5.9M, a large-scale benchmark comprising approximately 5.98 million images across six manipulation categories, including fully generated, partially edited, physically manipulated, and hybrid-edited images. The dataset adopts an operator-aligned generation strategy to reduce content-related shortcuts and encourage the learning of manipulation-specific forensic cues. The authors also propose ForgeFormer, a unified detection and localization framework based on a LoRA-adapted DINOv3 encoder and a Mask2Former-style decoder, incorporating multi-scale features, mask projection, supervised contrastive learning, and deep clustering. Extensive experiments on public benchmarks, unseen generators, and the independent IEDAL2 challenge demonstrate strong performance, including 87.7% balanced accuracy and 60.3% localization F1-score on IEDAL2.

Critical strengths
  1. The paper addresses an important limitation of existing forgery detection benchmarks by integrating fully generated, partially manipulated, physically edited, and hybrid-edited images into a unified dataset. Its panoptic provenance formulation is particularly meaningful, enabling different manipulation types to be distinguished within a single image rather than relying on binary localization. The large dataset scale, diverse manipulation operators, and operator-aligned construction strategy provide valuable resources for future research on generalizable image forensics.

  2. The proposed approach is evaluated across multiple manipulation categories, held-out generators, public benchmarks, and an independent challenge dataset. In particular, the IEDAL2 results provide valuable external evidence of generalization beyond the authors' data-generation pipeline. The component ablations further demonstrate the contributions of the panoptic formulation, mask projection, multi-scale feature processing, and auxiliary training objectives, supporting the practical value of the proposed framework.

Critical weaknesses
  1. Although operator alignment aims to reduce shortcut learning, the dataset still contains correlations between manipulation classes, source datasets, generators, and post-processing operations. For example, Appendix J reports that AI Inpainting Edit samples are exclusively drawn from VGG-Face2, while other classes use both VGG-Face2 and FaceForensics++. Additionally, the physical-edit pipeline applies JPEG recompression that is not matched across the generative branches. These differences may allow the model to exploit dataset-specific statistics rather than intrinsic forensic traces. Although the authors acknowledge these limitations, a controlled aligned-versus-unaligned comparison with matched data sources, generators, and post-processing is needed to establish the independent contribution of operator alignment.

  2. Table 2 compares approximately 1.79 million synthetic video frames against 31,783 real still images from Flickr30k, potentially introducing domain shortcuts related to image acquisition, compression, and processing. A source-matched evaluation would provide stronger evidence of cross-generator generalization. Furthermore, Table 3 reports a six-class Macro-F1 of 92.60% after removing the image-level classification loss, which is difficult to reconcile with Section 2.2.3, where this loss is described as the supervision for the six-class classifier. The authors should clarify how this classifier is trained and evaluated in the corresponding ablation.

Questions for authors
  1. Can the authors quantify the independent contribution of operator alignment through a controlled aligned-versus-unaligned experiment with matched dataset size, source composition, generator inventory, and post-processing?

  2. Why does Table 2 compare synthetic video frames with real still images from Flickr30k? Have the authors evaluated authentic and manipulated video frames matched in source, compression, resolution, and post-processing to exclude dataset-origin shortcuts?

  3. How is the image-level classifier trained in the "w/o image-level classification loss" ablation in Table 3? If the classification head receives no direct supervision, how is the reported 92.60% six-class Macro-F1 obtained?

  4. Since IEDAL2 evaluates binary forgery localization rather than hybrid-edit separation or multiclass panoptic attribution, what independent evidence supports the model's ability to distinguish co-occurring manipulation types outside the ForgePan-5.9M generation pipeline?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.