- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose a formalization of an issue in detecting edited regions, termed the "semantic plausibility shortcut" issue: a semantically plausible edit weakens the detection cues of the edited region compared to an abnormal one, which relates to six types of abnormalities (incompatibility). They introduce a bilingual benchmark called PAPE, where each sample is designed as a paired triplet consisting of a source image, a plausible edit, and an anomalous edit. This benchmark was curated via a two-stage pipeline. Finally, they provide a relative evaluation protocol (comparing plausible to abnormal IoU) and examine it across seven localizers.
Critical strengths
A key strength is the benchmark's paired design, which enables a highly controlled experimental setup. Through this, the authors expose an inherent semantic bias that previous datasets and localizers have failed to uncover.
Critical weaknesses
There are 4 main weaknesses:
1) Evaluation Protocol / Missing Experiments
- Limited diversity of source images (only natural images): The source images are restricted to natural domains (*COCO* and *ImageNet*). How do these findings transfer to animated or stylized content, where abnormal edits might appear more plausible due to the artistic style?
- Limitations of the relative IoU metric: The IoU is only calculated between the plausible and abnormal edits, rather than also measuring performance relative to the original source image (with no edited region).
- Lack of analysis on edit characteristics:
- How does performance vary based on edit sizes (small vs. large)?
- Does it matter if the edit is applied to the main foreground entity of the source image or a minor background element?
- How do different edit actions (e.g., removal, insertion, style modification) affect the results?
- Missing critical validation experiment: An important experiment is missing: evaluating the hypothesis on a set of real, non-edited "weird/anomalous" scenes to verify if the localizers behave differently without artificial edit artifacts.
2) Experimental Setup
- Unclear relevance of the bilingual setup: It is not clear how the bilingual setup relates to the primary experiment of this paper. What is the motivation for testing two languages? Do the underlying text encoders exhibit specific architectural or semantic differences that justify this?
- Unclear model selection criteria: What is the motivation behind choosing the four specific open-source models? Do they differ in interesting architectural ways that enrich the analysis?
- Absence of closed-source localizers: The paper does not evaluate any closed-source models, which could have been addressed by testing at least a small, representative subset.
3) Data Cases Coverage
- Lack of controlled prompts across image editors: As far as I understand, the prompts do not seem to be systematically processed across the four image editors (i.e., using the same prompt to generate four editor versions, multiplied by two for the plausible and abnormal pairs). If this is not the case, the setup is not sufficiently controlled, making it impossible to fairly compare the effects of different editors.
- Lack of analysis on abnormality levels: There is no systematic categorization or sorting of the data based on the "abnormality" level. Although the authors briefly mention in the results that the "Places" category is the easiest for detecting abnormalities, there is no organized profiling of abnormality levels in the dataset.
- Lack of manual verification for the curation pipeline: The curation pipeline lacks manual verification. A verified subset of the data is needed to confirm the quality of the pipeline, the dataset itself, and the correctness of the abnormal category assignments.
4) Missing Ablations
- Ablations on pipeline stages: The paper lacks ablation studies for stages 1 and 2 of the pipeline, specifically regarding:
- Mask generation (SIGMA)
- Threshold selections
- Prompt validation criteria
Questions for authors
Additional Questions for the Authors
- Unclear value-add of the bilingual setup: How exactly does the bilingual setup contribute to the findings or the benchmark? What is the relationship or correlation between the language of the prompt and the "semantic plausibility shortcut" issue? Do the text encoders exhibit language-specific biases?
- Outdated evaluation suite: Four out of the six localizers evaluated in the paper are from the 2021–2023 period. Why did the authors not include more recent state-of-the-art localizers from 2024 to 2026? Are they the SOTA today?
Writing and Figures Notes
-
Incoherent text and poor flow: The overall writing is quite disjointed and lacks smooth transitions. This poor narrative flow is particularly noticeable in the Introduction. The introduction suffers from a lack of proper motivation and clear definitions before diving directly into the results (particularly in the second paragraph). The reader needs a stronger foundation regarding the core problem before seeing the findings.
-
Descriptive instead of Analytical Dataset Statistics: The dataset statistics section (lines 300-311) merely lists raw, technical observations. It reads like a pure technical report rather than drawing any higher-level insights or scientific claims from the data.
-
Confusing Experimental Structure: The organization of the experimental setup and the experiments themselves is messy and hard to follow. It would be much clearer to categorize the evaluated localizers by their technical approach (e.g., architecture, objective).
-
Technical-Report Style Results Section: The experimental results section reads like a dense technical report. It consists of three large tables and raw observations without sufficient synthesis.
-
Weak Related Work and Architectural Context: The related work regarding localizer differences is poorly explained. The authors rely on general titles and vague distinctions, which makes it difficult to understand the underlying causes of the experimental results.
-
Missing Limitations, Future Work, or Mitigation Strategies: The paper does not mention any limitations or potential directions for future work. More importantly, it lacks any concrete proposals, suggestions, or guidelines on how to actually mitigate this semantic bias in future localizers.
-
Unclear Abstract phrasing: The abstract's presentation is saturated with overly complex or ambiguous terms. It needs to be simplified to clearly convey the core contribution.
-
Overuse of Alphabet Soup: The paper relies heavily on excessive abbreviations and acronyms (e.g., SPPV, SSTOA), which significantly hurts readability and clutters the text.
-
Potential error in Figure 1: The ground-truth (GT) masks shown for the "plausible" and "anomalous" edits appear to be identical. Is this a presentation error? Please clarify.
-
Unclear labeling in Figure 3 (c): What exactly does the target group labeled "artifacts" represent in this context? A clearer definition or explanation is needed.
7 comments
Response to Reviewer a8dv— Part 2 of 5: Edit Characteristics, Non-edited Images, and Experimental Setup
Response to Weakness #1.3
2. Foreground/background location. The dataset does not provide strict foreground/background annotations, so we define a proxy using the prompt and edit-mask position. An unambiguous foreground or background reference in the prompt determines the label. Otherwise, we assign the foreground label when the plausible-mask centroid lies within the central 50% of both image dimensions (25%–75%), and the background label otherwise. Tables R3(a)–R3(c) show higher anomalous-edit IoU in 19 of 21 foreground-proxy and all 21 background-proxy model–dataset combinations. Thus, the semantic-plausibility gap appears in both location groups and is generally more pronounced in the background group.
Table R3(a). Localization performance by foreground/background proxy on COCO (foreground 80.9%, background 19.1%).
| Method | Foreground | Background |
|---|---|---|
| SIDA-7B | 0.323 | 0.652 |
| ForgeryTalker | 0.209 | 0.359 |
| ObjectFormer | 0.092 | 0.383 |
| IML-ViT | 0.079 | 0.371 |
| CAT-Net | −0.179 | 0.204 |
| PSCC-Net | −0.003 | 0.312 |
| TruFor | 0.203 | 0.367 |
Table R3(b). Localization performance by foreground/background proxy on ImageNet (foreground 89.8%, background 10.2%).
| Method | Foreground | Background |
|---|---|---|
| SIDA-7B | 0.330 | 0.541 |
| ForgeryTalker | 0.187 | 0.156 |
| ObjectFormer | 0.093 | 0.250 |
| IML-ViT | 0.096 | 0.262 |
| CAT-Net | 0.046 | 0.333 |
| PSCC-Net | 0.006 | 0.226 |
| TruFor | 0.145 | 0.282 |
Table R3(c). Localization performance by foreground/background proxy on Places (foreground 78.0%, background 22.0%).
| Method | Foreground | Background |
|---|---|---|
| SIDA-7B | 0.406 | 0.519 |
| ForgeryTalker | 0.234 | 0.253 |
| ObjectFormer | 0.245 | 0.377 |
| IML-ViT | 0.252 | 0.326 |
| CAT-Net | 0.296 | 0.461 |
| PSCC-Net | 0.235 | 0.341 |
| TruFor | 0.315 | 0.357 |
3. The results of different edit actions (e.g., removal, insertion, style modification). We use replacement for both plausible and anomalous edits to hold the edit operation fixed and focus the analysis on semantic plausibility. Extending the paired evaluation to these operations is a direction for future work.
Response to Weakness #1.4
1. Dataset construction. We collected candidate photographs from public photography communities through keyword searches and photography groups, focusing on unusual apparent scales or interactions created by perspective, occlusion, and camera viewpoint. We manually inspected the images and their source pages, excluded weak anomalous relationships and obvious AI-generated or composited images, and removed exact and near duplicates using SHA-256 and perceptual hashes. Representative images and their localization visualizations will be added to the paper's appendix.
2. Localization performance. Since these images have no edited region, we measure false-positive rates. Pixel false-positive rate is the fraction of output-grid pixels exceeding threshold , averaged equally over images; image false-positive rate is the proportion of images with at least one such pixel. Table R4 reports all nine localizers. At , mean false-positive pixel area ranges from 0.37% for SIDA-7B to 11.99% for ObjectFormer, with 11.52% for FMAE and 11.17% for PSCC-Net.
These results show that localization models can flag non-edited anomalous scenes as manipulated, including at stricter prediction thresholds.
Table R4. False positives on 500 non-edited anomalous images.
| Localizer | Pixel FP, 0.5 | Pixel FP, 0.9 | Image FP, 0.5 |
|---|---|---|---|
| SIDA-7B | 0.37% | 0.11% | 1.8% |
| ForgeryTalker | 6.69% | 0.70% | 100.0% |
| ObjectFormer | 11.99% | 10.00% | 94.0% |
| IML-ViT | 3.96% | 1.87% | 100.0% |
| SparseViT | 1.49% | 0.33% | 63.2% |
| FMAE | 11.52% | 3.17% | 99.6% |
| CAT-Net | 1.85% | 0.63% | 61.4% |
| PSCC-Net | 11.17% | 3.11% | 97.0% |
| TruFor | 1.77% | 0.89% | 46.6% |
Joint response to Weakness #2.1 and Additional Question #1
We use Chinese and English to broaden instruction coverage and provide two traceable language realizations of the same intended semantic contrast. Each plausible–anomalous comparison is performed within the same language, keeping prompt language fixed when measuring the localization gap. Tables 4–6 pool both languages and quantify the plausible–anomalous gap across the bilingual collection. A separate language-stratified evaluation would quantify the relationship between prompt language and the gap, with controlled encoder analyses needed to attribute any language-specific differences to the text encoder.
Response to Reviewer a8dv— Part 3 of 5: Closed-source Evaluation, Prompt Controls, and Violation Types
Response to Weakness #2.2
We selected Qwen-Image-Edit-2511, FireRed-Image-Edit-1.1, LongCat-Image-Edit, and FLUX.2-dev because they provide public, reproducible checkpoints, support instruction-guided replacement, and can follow the same paired prompts under controlled generation settings. Their architectures also vary in backbone scale and multimodal processing: Qwen is a double-stream DiT; FireRed adds 3D RoPE; LongCat is a 6B hybrid; FLUX.2-dev is a 32B rectified-flow model.
Response to Weakness #2.3
Thank you for suggesting an evaluation of a closed-source localizer. We added TextIn, a commercial image-manipulation detection service from INTSIG (合合信息), and compared it with all nine public checkpoints on the same randomly sampled 500-pair subset.
Evaluate closed-source model. The subset contains 167 COCO, 167 ImageNet, and 166 Places pairs, covering the four editors and six violation types. We convert TextIn's returned PS-tamper regions and confidence scores into localization maps, then evaluate them at threshold 0.5 against each edited image's SIGMA mask. As shown in Table R5, TextIn obtains mean IoUs of 0.077 on plausible edits and 0.129 on anomalous edits, with a relative gap of 0.403. All ten localizers have positive gaps on this subset.
Table R5. Closed- and open-source localization on the same 500-pair audit subset.
| Localizer | |
|---|---|
| SIDA-7B | 0.750 |
| ForgeryTalker | 0.366 |
| ObjectFormer | 0.250 |
| IML-ViT | 0.205 |
| SparseViT | 0.203 |
| FMAE | 0.089 |
| CAT-Net | 0.197 |
| PSCC-Net | 0.115 |
| TruFor | 0.357 |
| TextIn | 0.403 |
Response to Weakness #3.1
All four editors receive exactly the same prompts. For each source image and language, we generate one plausible–anomalous instruction pair and apply it unchanged to Qwen, FireRed, LongCat, and FLUX.2. Within each editor, the two conditions use the same generation settings. The final release aligns 6,500 source photographs across all four editors. Differences in retained-pair counts arise from editor-specific mask-overlap filtering and the retention of Chinese and English variants, while the input prompts remain identical across editors.
Response to Weakness #3.2
We characterize abnormality through six semantic-violation types: C1 physical attribute, C2 function or affordance, C3 biological structure, C4 spatial relation, C5 interaction relation, and C6 scene context. Tables R6(a)–R6(c) now report relative gap separately for each violation type and source dataset, averaged equally across the four editors. The gap is positive in 37/42 COCO, 38/42 ImageNet, and 42/42 Places model–category comparisons, totaling 117/126. These results show that lower localization performance on plausible edits occurs across all six violation types, supporting a semantic-plausibility shortcut shared by multiple localizers across different forms of semantic conflict.
Table R6(a). Localization performance by violation type on COCO.
| Method | C1 | C2 | C3 | C4 | C5 | C6 |
|---|---|---|---|---|---|---|
| SIDA-7B | 0.117 | 0.867 | 0.535 | 0.415 | 0.405 | −0.145 |
| ForgeryTalker | 0.276 | 0.162 | 0.228 | 0.213 | 0.153 | 0.210 |
| ObjectFormer | 0.164 | 0.130 | 0.115 | 0.024 | 0.216 | 0.282 |
| IML-ViT | 0.135 | 0.316 | 0.145 | 0.017 | 0.326 | −0.039 |
| CAT-Net | −1.041 | 0.305 | −0.260 | 0.220 | 0.072 | 0.125 |
| PSCC-Net | 0.008 | 0.155 | 0.107 | 0.114 | −0.361 | 0.157 |
| TruFor | 0.272 | 0.195 | 0.234 | 0.643 | 0.033 | 0.269 |
Table R6(b). Localization performance by violation type on ImageNet.
| Method | C1 | C2 | C3 | C4 | C5 | C6 |
|---|---|---|---|---|---|---|
| SIDA-7B | −0.025 | 0.335 | 0.498 | 0.394 | 0.352 | 0.414 |
| ForgeryTalker | 0.115 | 0.142 | 0.106 | 0.295 | 0.230 | 0.238 |
| ObjectFormer | 0.060 | 0.138 | 0.021 | 0.140 | −0.051 | 0.327 |
| IML-ViT | 0.098 | 0.100 | 0.027 | 0.206 | 0.088 | 0.214 |
| CAT-Net | 0.055 | 0.074 | 0.012 | 0.188 | 0.130 | 0.151 |
| PSCC-Net | −0.052 | −0.035 | 0.018 | 0.097 | 0.041 | 0.117 |
| TruFor | 0.173 | 0.120 | 0.064 | 0.329 | 0.130 | 0.205 |
Table R6(c). Localization performance by violation type on Places.
| Method | C1 | C2 | C3 | C4 | C5 | C6 |
|---|---|---|---|---|---|---|
| SIDA-7B | 0.176 | 0.648 | 0.575 | 0.238 | 0.363 | 0.497 |
| ForgeryTalker | 0.160 | 0.241 | 0.175 | 0.363 | 0.288 | 0.181 |
| ObjectFormer | 0.100 | 0.321 | 0.322 | 0.292 | 0.201 | 0.431 |
| IML-ViT | 0.077 | 0.354 | 0.278 | 0.385 | 0.307 | 0.287 |
| CAT-Net | 0.084 | 0.324 | 0.419 | 0.493 | 0.480 | 0.257 |
| PSCC-Net | 0.055 | 0.294 | 0.300 | 0.360 | 0.290 | 0.275 |
| TruFor | 0.233 | 0.350 | 0.361 | 0.346 | 0.386 | 0.314 |
Response to Reviewer a8dv— Part 1 of 5: Stylized Images, Localization Metrics, and Edit Size
Response to Weakness #1.1
Stylized-image construction. We randomly sampled 1,000 images from
norod78/cartoon-blip-captionsand ran the same prompt-generation, editing, mask-generation, and overlap-filtering pipeline. Qwen-Image-Edit generated the corresponding 3,956 edited images; 1,341 pairs passed the pair-mask IoU ≥0.5 filter.Localization results. We evaluated all seven localizers on these 1,341 pairs. Table R1 shows higher IoU on anomalous edits for every method, with relative gaps from 0.081 to 0.572. The gap therefore extends to this cartoon corpus under Qwen editing.
Table R1. Localization performance on the stylized-image dataset.
Response to Weakness #1.2
1. Purpose of the relative IoU metric. For each edited condition z∈pl,an, we compute Jiz=IoU(M^iz,Miz) against that output's own edit mask. We then measure the performance difference as Δrel=(Jˉan−Jˉpl)/Jˉan, where a positive value indicates lower localization IoU on plausible edits. This paired metric compares the two edited conditions and it does not evaluate an unedited image, which has no manipulated region to localize.
2. False-positive rates on unedited images. With an empty ground-truth manipulation mask, any pixel predicted as manipulated is a false positive. We report both measures on 500 non-edited anomalous images (Table R4 in Response to Weakness #1.4).
Response to Weakness #1.3
1. Performance vary based on edit sizes. For condition z∈pl,an, we define edited-region size as the fraction of image pixels covered by its binary SIGMA mask:
Aiz=HiWi∣Miz∣,
where ∣Miz∣ is the number of foreground pixels in the mask and HiWi is the image area. For paired localization, both members receive the same size label, determined by their mean area ratio:
Aˉi=2Aipl+Aian.
We define small regions by Aˉi<0.05, medium regions by 0.05≤Aˉi<0.20, and large regions by Aˉi≥0.20. We re-aggregate the official-test IoUs within each size bin, averaging within each condition and editor before taking an unweighted mean across the four editors. Tables R2(a)–R2(c) report relative gap Δrel for COCO, ImageNet, and Places. The gap is positive in all 21 small-region and all 21 medium-region model–dataset combinations, and in 18 of 21 large-region combinations, totaling 60 of 63 (Tables R2(a)–R2(c)). The semantic-plausibility gap therefore appears across region sizes for multiple models.
Table R2(a). Localization performance by edit size on COCO (small 6.0%, medium 38.1%, large 55.9%).
Table R2(b). Localization performance by edit size on ImageNet (small 2.7%, medium 25.0%, large 72.2%).
Table R2(c). Localization performance by edit size on Places (small 6.2%, medium 36.0%, large 57.7%).