- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper tackles the problem of localising image edits in instruction-controlled image editing via generative models. There are two main contributions: a) a new image-editing benchmark, EditScan-bench; and b) EditScan, a baseline method for detecting image edits. The novelty of the benchmark is that it extends beyond simple crop/splicing setups or inpainting benchmarks. The novelty of the method is a two stage architecture that combines global, image-level detection with local patchwise detection to enhance identification of pixel-level and image-level edits. While the performance results are good, I have concerns regarding the evaluation rigour. Specifically, the method appears to not fairly disentangle the performance gains stemming from the benchmark against those stemming from the detection method, prompting me to give it a reject.
Critical strengths
- The paper tackles a very relevant problem, e.g., detecting ai-modified images is very relevant for countering deep-fakes.
- The paper is generally well
Critical weaknesses
-
It is not clear if the baseline detection methods were trained on the same EditScan-Bench. The description in Sec. 5.1 only says that all baselines were evaluated on the same test set. The code in the github repository lacks the scripts required to train the baselines on the proposed EditScan-Bench dataset, leading me to believe that those methods are applied off-the-shelf using their pre-loaded checkpoints during the evaluation. If this is the case, the performance results are unfair to the baselines, since it does not indicate whether the better results of EditScan stem from training on a different dataset or from the methodological superiority.
-
Continuing the above weakness, the paper does not separate the gains stemming from the new Benchmark, versus the methodological gains on the detection model itself. An ablation study comparing the OOD performance of EditScan’s (and the baselines’) when trained on an existing benchmark dataset against a setting where all methods train on the EditScan-Bench, is necessary to isolate the effect of the new benchmark.
-
The paper uses the DinoV3 model as a backbone in the detection pipeline. However, it appears that the baseline methods do not use the same backbone, which adds an additional confounding layer. It is not clear if the performance gains stem from using a better backbone or from the loss function and other proposed designs.
-
The paper claims that the benchmark was validated by human experts. However there are insufficient details regarding how this validation was conducted. For example, there is a lack of information on the number of annotators, how annotations across subjects were combined, whether there were labelling differences between annotators, how they were selected, etc.
-
Questions for authors
- Can you provide the exact details on how the baselines were trained? Specifically, were they fine-tuned on EditScan-Bench when reporting the results, or were they pre-trained on their own detection benchmarks?
- Can you provide details on how the human evaluation was conducted? Preview