acceptodds
Under review as a conference paper at ICLR 2027

Evolving Multi-modal Large Language Model Without Training for Zero-shot Image Forgery Localization

Abstract

Image Forgery Localization(IFL) aims to identify and precisely locate tampered regions in images. To tackle this practical but challenging task, existing methods typically detect forged region in different scenarios by employing a large amount of pre-collected training data. However, with the rapid iteration and upgrading of forgery strategies, new forgery localization requirements are often accompanied by training data that cannot be collected in advance, making it difficult for existing methods to fail against the latest forgery techniques. In this article, we focus on a such practical and challenging task, namely Zero-shot Image Forgery Localization(ZIFL). Inspired by recent advances in Multi-modal Large Language Models(MLLMs), we propose a training-free image forgery localization method called Collaborative LEAder-worker discrimiNator, called CLEAN, which performs zero-shot forgery localization by gathering the cross-modal reasoning capacity of MLLMs and the fine-grained localization capacity of Segment Anything(SAM) to identify forgery clues. Specifically, we propose a Coarse-Grained Suspect Exploration(CGSE) module to act as a forensic analysis leader and systematically generate a list of candidate suspect regions based on high-level forgery evidence. Besides, a Fine-Grained Iterative Refinement(FGIR) module is designed to judge the forgery clues in an iterative manner, which eliminates normal backgrounds and refines the boundaries of tampered regions for accurate ZIFL. Extensive experiments on multiple benchmarks demonstrate that our method achieves competitive IFL performance against existing training-based methods without relying on pre-collected training data on zero-shot image forgery localization task. Our code will be released soon.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
image forgery localization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_yqTtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper targets the core limitation of traditional Image Forgery Localization (IFL) methods, which heavily rely on task-specific training data and suffer from poor generalization to emerging and unseen forgery manipulations (e.g., GAN editing, diffusion inpainting, and new MLLM-based image tampering). The authors formally define the Zero-shot Image Forgery Localization (ZIFL) task, which aims to achieve pixel-level forgery localization on unknown manipulation types without any task-specific training or parameter fine-tuning.

To address this task, the paper proposes a Collaborative LEAder-worker discrimiNator (CLEAN) framework, which innovatively combines frozen pre-trained Multimodal Large Language Models (MLLMs) and LangSAM. The core insight is to convert traditional low-level forensic artifact detection into prior-violation analysis: leveraging MLLMs’ learned visual, semantic, and physical world priors to identify inconsistent and anomalous content in forged images, rather than relying on manually defined forgery traces.

The CLEAN framework consists of two core modules. First, the Coarse-Grained Suspect Exploration (CGSE) module generates structured suspicious region proposals via multi-dimensional forensic reasoning (semantic consistency, digital integrity, and physical-environment consistency), and adopts a context-aware bounding box expansion strategy to avoid missing forgery boundaries. Second, the Fine-Grained Iterative Refinement (FGIR) module verifies and optimizes initial proposals through confidence pre-judgment, leader-worker clue decomposition, LangSAM segmentation, and critic-guided iterative prompt refinement, finally outputting accurate pixel-level forgery masks.

Extensive experiments are conducted on 7 mainstream IFL benchmarks covering both traditional Photoshop-based forgery and AIGC-generated tampering, with six representative MLLMs adopted for evaluation. The results demonstrate that CLEAN achieves consistent and significant performance gains in IoU and F1 scores over vanilla MLLM baselines, and even outperforms some supervised training-based SOTA IFL methods. Ablation studies further validate the indispensable roles of the CGSE proposal generation and FGIR refinement modules, as well as the effectiveness of the three-dimensional forensic reasoning dimensions.

Critical strengths
  • Meaningful and practical problem formulation: The ZIFL task defined in this paper is of high research and application value. It effectively addresses the critical pain point of traditional supervised IFL models, i.e., severe generalization degradation when facing out-of-distribution and newly emerging forgery manipulations caused by the rapid iteration of generative image editing technologies. This problem definition is more forward-looking than incremental architectural improvements of existing IFL methods.

  • Reasonable and interpretable cross-model collaboration paradigm: The paper designs a clear logical division of labor between MLLMs and SAM-based segmentation models: MLLMs undertake high-level forensic reasoning, suspicious region mining, and clue verification, while LangSAM is responsible for precise pixel-level localization. Different from simple model stacking, the proposed pipeline forms a closed-loop system ofreasoning-evidence-segmentation-refinement, which effectively bridges the gap between open-ended visual anomaly reasoning and precise segmentation localization.

  • Innovative coarse-to-fine iterative refinement mechanism: The FGIR module is the core contribution of the system design. The leader-worker decomposition strategy successfully converts abstract forensic anomaly descriptions into concrete visual segmentation prompts, building a reliable semantic bridge between forensic evidence and visual entity segmentation. Moreover, the critic-guided boundary evolution mechanism iteratively optimizes segmentation masks by checking evidence completeness and boundary rationality, effectively solving the mismatch between natural object contours and real forgery boundaries.

  • Comprehensive and robust experimental validation: The experimental setup is rigorous and sufficient. The method is validated on six diverse MLLM backbones and seven benchmarks covering both traditional manual forgery and cutting-edge AIGC tampering scenarios. Consistent performance improvements across all settings prove that CLEAN’s effectiveness is not a backbone-specific prompt trick, reflecting good generalizability. In addition, detailed ablation experiments on core modules and forensic reasoning dimensions fully verify the rationality of each design component.

  • Insightful prior-violation analysis perspective: The core observation of replacing traditional low-level forensic artifact detection with MLLM prior-violation analysis is the most valuable conceptual insight of this work. It provides a novel paradigm for zero-shot IFL by exploiting the inherent visual, semantic and physical knowledge of pre-trained MLLMs, rather than relying on task-specific supervised learning.

Critical weaknesses
  • Overly loose and controversial zero-shot definition: The paper claims a pure zero-shot setting, but the setting is essentiallytraining-free/task-adaptation-free rather than genuine zero-shot forgery localization. All adopted MLLMs and LangSAM are powerful pre-trained foundation models that have already absorbed massive internet data and potential knowledge of AIGC generation models involved in test datasets (e.g., GPT-4o, Diffusion models). The claim that the model has never seen unseen manipulations is not strictly proven, which overstates the technical contribution.
  • Core performance heavily relies on manual prompt engineering: The so-called forensic reasoning capability of MLLMs is essentially driven by manually designed forensic prompt checklists (including explicit definitions of semantic, digital, and physical anomalies such as unnatural edges, texture discontinuity, and inconsistent lighting). The method injects prior forensic domain knowledge into MLLMs via handcrafted prompts, rather than mining inherent forensic awareness from MLLM pre-trained representations. This weakens the scientific novelty of the work.
  • Limited algorithmic innovation (mostly agentic prompting pipeline): The proposed CLEAN framework has no trainable parameters or novel network architectures. The leader-worker decomposition, evidence verification, and critic refinement modules are essentially multi-stage, structured MLLM prompt orchestration and iterative inference. From the algorithmic novelty perspective, this work is a well-designed agentic reasoning pipeline rather than a novel learning-based IFL algorithm, which is a critical limitation for top conference acceptance. Unconvincing theoretical justification: The theoretical convergence analysis in the appendix is based on an unvalidated core assumption (0 < ρ < 1). The paper assumes that critic feedback can continuously reduce localization errors, but there is no experimental evidence to support this assumption. In practice, the critic mechanism may fail or even degrade mask quality due to incorrect prior evidence, making the theoretical guarantee conditional and superficial.
  • Unresolved fundamental mechanism gap in forensic localization: The paper fails to explore the internal mechanism of how MLLMs perceive low-level forensic residuals (noise, edge, texture anomalies). As semantic-dominant visual models, MLLMs are inherently insensitive to signal-level subtle variations, and the improvement purely relies on customized prompts. There is a lack of feature-level, frequency-domain analysis and residual visualization experiments to verify the reliability of the transformation from low-level forensic residuals to high-level semantic anomalies.
  • Cascading error propagation and confirmation bias risks: The entire pipeline suffers from severe error accumulation problems. Incorrect suspicious region proposals and wrong forensic clues generated in the CGSE stage will be continuously verified and amplified by the worker modules, with no independent error correction mechanism. The critic module only checks mask evidence coverage instead of re-judging the authenticity of forgery clues, which cannot rectify initial reasoning errors and leads to inherent confirmation bias.
Questions for authors

Please refer to the weaknesses.

Reviewer_Ss22OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses zero-shot image forgery localization using frozen multimodal large language models and LangSAM. The proposed CLEAN framework contains two stages. Coarse-Grained Suspect Exploration identifies candidate regions using semantic, digital-integrity, and physical-consistency clues, then expands their bounding boxes to retain context. Fine-Grained Iterative Refinement verifies candidate evidence, generates segmentation prompts through a Leader-Worker mechanism, and uses Critic feedback to refine local masks before merging them. The paper evaluates the framework across seven benchmarks and six MLLM backbones, with component ablations and efficiency analysis. Its central aim is to connect high-level forensic reasoning with pixel-level localization without task-specific parameter updates.

Critical strengths
  1. A useful training-free formulation. Combining frozen MLLMs with segmentation models could reduce dependence on task-specific annotations and retraining as manipulation techniques evolve. The separation between suspect exploration, evidence verification, and mask refinement is a reasonable design for connecting semantic reasoning to pixel-level localization (Sections 3.3-3.4, pp. 4-6).

  2. Relevant component analyses and implementation material. The paper includes module-removal ablations, evidence-dimension analysis, iteration-count experiments, inference timings, and detailed prompt templates (Figure 3, p. 8; Table 3 and Figure 6, p. 17; Table 4, p. 18; Appendix I, pp. 19-24). These materials make the design inspectable and provide preliminary support for its components, subject to the evaluation inconsistencies below.

Critical weaknesses
  1. W1: Metric inconsistencies leave the main empirical conclusions unresolved. Under Equations (34) and (37), identical predictions, samples, and averaging weights with consistent empty-mask conventions require mean IoU <= mean F1 <= 2(mean IoU)/(1+mean IoU) (p. 16). Tables 1-2 violate these bounds: MPC on IMD2020 reports IoU/F1 of 0.48/0.40, while LLaVA-1.5+CLEAN on NIST16 reports 0.24/0.45, exceeding the upper bound of approximately 0.387 (p. 7). These discrepancies exceed rounding error. Different evaluation conventions might explain the entries, but Appendix C does not specify the differences needed to reconcile them. The claimed gains require verifiable metric calculations or corrections; the discrepancies alone do not establish their cause.

  2. W2: The contribution of refinement is difficult to isolate and reproduce. Equation (5) and Algorithm 1 use lambda only to sort ROIs before separate processing and mask union (pp. 5 and 15). With fixed proposals and deterministic, independent processing, ordering cannot affect the result, yet Figure 5(b) varies with lambda (p. 17). The relevant implementation dependency or uncontrolled variation is unexplained. Baseline and module-removal mask generation are also insufficiently specified (pp. 8-9), obscuring the contributions of segmentation, cropping, and refinement. Figure 6 provides iteration-count evidence, but attribution requires clarification of the components and initial conditions held fixed, alongside the corresponding inference cost (pp. 17-18).

  3. W3: The geometric coverage argument does not follow from the stated assumption. Equation (15) bounds deviation by alpha times max(w,h), whereas Equation (16) expands the box by alphaw horizontally and alphah vertically (p. 13). With w=10, h=100, and alpha=0.1, the bound permits a horizontal deviation of 10 pixels, but expansion adds only 1 pixel per side. Containment and unit recall in Equations (17)-(18) therefore need stronger coordinate-specific assumptions or a revised expansion rule. This is a local flaw in the theoretical justification, rather than evidence that the practical framework cannot work.

Questions for authors
  1. Can evaluation code or prediction-level calculations reconcile the incompatible pairs in Tables 1-2? Please specify whether both metrics use the same predictions, samples, averaging weights, and empty-mask conventions, and show whether the main comparative conclusions survive any corrections (pp. 7 and 16).

  2. Besides sorting ROIs, what does lambda affect? What order-dependent operation or uncontrolled variation explains Figure 5(b), given Algorithm 1's separate ROI processing and union operation (pp. 15 and 17)?

  3. How do the direct MLLM baseline and module-removal variants generate pixel masks? In Figure 6's one-versus-three-iteration comparison, are the MLLM, LangSAM, candidate regions, and initial prompts held fixed, and what is the corresponding inference cost (pp. 8-9 and 17-18)?

  4. How does Equation (15) imply containment in Equation (17) for non-square boxes? Please correct the deviation assumption or expansion rule and state the conditions under which unit recall follows (Appendix A.1, p. 13).

Reviewer_YB3oOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a training-free framework named CLEAN for zero-shot image forgery localization. It combines a multimodal large language model (MLLM) with the SAM‑family segmentation model in two stages: Coarse-Grained Suspect Exploration (CGSE) and Fine-Grained Iterative Refinement (FGIR). CGSE generates candidate regions and forensic clues using semantic, digital-integrity, and physical-consistency reasoning. FGIR verifies these clues and iteratively refines segmentation masks. Then, CLEAN is evaluated with six different MLLM backbones across seven benchmarks.

Critical strengths
  1. The design connects complementary capabilities. CLEAN uses MLLMs to propose and reason about suspicious regions, then uses LangSAM to produce pixel-level masks.

  2. The evaluation includes both traditional and AIGC-based manipulations.

Critical weaknesses
  1. The paper reports ablations of CGSE and FGIR as modules, but it is harder to isolate the contribution of individual FGIR steps.

  2. The paper provides qualitative examples of successful cases but lacks systematic analysis of failure samples.

  3. Current benchmarks mostly contain one or a small number of large forged regions. Real‑world forgeries may consist of many tiny, scattered tampered patches.

  4. If several ROIs contain small false‑positive segmentation outputs, union operation aggregates these errors and raises overall false‑positive rate. The paper does not analyse this risk.

Questions for authors
  1. The final mask uses simple union of ROI masks. Have you explored alternative fusion schemes, such as confidence‑weighted fusion or voting?

  2. How does CLEAN behave for images containing many small scattered forged patches?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.