acceptodds
Withdrawn from ICLR 2027

Can You Find the Edit When Nothing Looks Wrong? A Paired Benchmark for Manipulation Localization Beyond Semantic Abnormality

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

Instruction-driven image editing can produce manipulations that remain semantically coherent with the surrounding scene. Standard per-image evaluations do not reveal whether a localizer detects forensic evidence or relies on semantic abnormality that coincides with the edited region. We formalize this issue as the semantic-plausibility shortcut: a model may localize an anomalous replacement yet miss a plausible edit of the same target. We introduce Plausible–Anomalous Paired Edits (PAPE), a benchmark that makes intended semantic plausibility an explicit evaluation factor. Each triplet contains a source image and two edits of the same target, one plausible and one anomalous, produced with one editor and matched controlled editing settings, independently annotated masks, and overlap filtering for comparable spatial support. A two-stage construction pipeline yields ∼47K bilingual triplets from 6.5K photographs shared across four editors, three source domains, and six semantic violation categories. A pair-preserving protocol reports localization quality in each condition and the gap between them. Across seven public localizers, editor-averaged IoU is lower on plausible edits for every method on ImageNet and Places and for six of seven on COCO, with positive relative gaps from 2.5% to 42.2%. PAPE exposes a reliability dimension that aggregate scores conceal and provides paired records for analyzing its causes. Our anonymized code and dataset is available at https://anonymous.4open.science/r/PAPEDataset/.

Primary area:
datasets and benchmarks
Keywords:
image manipulation localization, semantic plausibility, image forensics, instruction-driven image editing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_a8dvOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a formalization of an issue in detecting edited regions, termed the "semantic plausibility shortcut" issue: a semantically plausible edit weakens the detection cues of the edited region compared to an abnormal one, which relates to six types of abnormalities (incompatibility). They introduce a bilingual benchmark called PAPE, where each sample is designed as a paired triplet consisting of a source image, a plausible edit, and an anomalous edit. This benchmark was curated via a two-stage pipeline. Finally, they provide a relative evaluation protocol (comparing plausible to abnormal IoU) and examine it across seven localizers.

Critical strengths

A key strength is the benchmark's paired design, which enables a highly controlled experimental setup. Through this, the authors expose an inherent semantic bias that previous datasets and localizers have failed to uncover.

Critical weaknesses

There are 4 main weaknesses:

1) Evaluation Protocol / Missing Experiments

  • Limited diversity of source images (only natural images): The source images are restricted to natural domains (*COCO* and *ImageNet*). How do these findings transfer to animated or stylized content, where abnormal edits might appear more plausible due to the artistic style?
  • Limitations of the relative IoU metric: The IoU is only calculated between the plausible and abnormal edits, rather than also measuring performance relative to the original source image (with no edited region).
  • Lack of analysis on edit characteristics:
    • How does performance vary based on edit sizes (small vs. large)?
    • Does it matter if the edit is applied to the main foreground entity of the source image or a minor background element?
    • How do different edit actions (e.g., removal, insertion, style modification) affect the results?
  • Missing critical validation experiment: An important experiment is missing: evaluating the hypothesis on a set of real, non-edited "weird/anomalous" scenes to verify if the localizers behave differently without artificial edit artifacts.

2) Experimental Setup

  • Unclear relevance of the bilingual setup: It is not clear how the bilingual setup relates to the primary experiment of this paper. What is the motivation for testing two languages? Do the underlying text encoders exhibit specific architectural or semantic differences that justify this?
  • Unclear model selection criteria: What is the motivation behind choosing the four specific open-source models? Do they differ in interesting architectural ways that enrich the analysis?
  • Absence of closed-source localizers: The paper does not evaluate any closed-source models, which could have been addressed by testing at least a small, representative subset.

3) Data Cases Coverage

  • Lack of controlled prompts across image editors: As far as I understand, the prompts do not seem to be systematically processed across the four image editors (i.e., using the same prompt to generate four editor versions, multiplied by two for the plausible and abnormal pairs). If this is not the case, the setup is not sufficiently controlled, making it impossible to fairly compare the effects of different editors.
  • Lack of analysis on abnormality levels: There is no systematic categorization or sorting of the data based on the "abnormality" level. Although the authors briefly mention in the results that the "Places" category is the easiest for detecting abnormalities, there is no organized profiling of abnormality levels in the dataset.
  • Lack of manual verification for the curation pipeline: The curation pipeline lacks manual verification. A verified subset of the data is needed to confirm the quality of the pipeline, the dataset itself, and the correctness of the abnormal category assignments.

4) Missing Ablations

  • Ablations on pipeline stages: The paper lacks ablation studies for stages 1 and 2 of the pipeline, specifically regarding:
    • Mask generation (SIGMA)
    • Threshold selections
    • Prompt validation criteria
Questions for authors

Additional Questions for the Authors

  • Unclear value-add of the bilingual setup: How exactly does the bilingual setup contribute to the findings or the benchmark? What is the relationship or correlation between the language of the prompt and the "semantic plausibility shortcut" issue? Do the text encoders exhibit language-specific biases?
  • Outdated evaluation suite: Four out of the six localizers evaluated in the paper are from the 2021–2023 period. Why did the authors not include more recent state-of-the-art localizers from 2024 to 2026? Are they the SOTA today?

Writing and Figures Notes

  • Incoherent text and poor flow: The overall writing is quite disjointed and lacks smooth transitions. This poor narrative flow is particularly noticeable in the Introduction. The introduction suffers from a lack of proper motivation and clear definitions before diving directly into the results (particularly in the second paragraph). The reader needs a stronger foundation regarding the core problem before seeing the findings.

  • Descriptive instead of Analytical Dataset Statistics: The dataset statistics section (lines 300-311) merely lists raw, technical observations. It reads like a pure technical report rather than drawing any higher-level insights or scientific claims from the data.

  • Confusing Experimental Structure: The organization of the experimental setup and the experiments themselves is messy and hard to follow. It would be much clearer to categorize the evaluated localizers by their technical approach (e.g., architecture, objective).

  • Technical-Report Style Results Section: The experimental results section reads like a dense technical report. It consists of three large tables and raw observations without sufficient synthesis.

  • Weak Related Work and Architectural Context: The related work regarding localizer differences is poorly explained. The authors rely on general titles and vague distinctions, which makes it difficult to understand the underlying causes of the experimental results.

  • Missing Limitations, Future Work, or Mitigation Strategies: The paper does not mention any limitations or potential directions for future work. More importantly, it lacks any concrete proposals, suggestions, or guidelines on how to actually mitigate this semantic bias in future localizers.

  • Unclear Abstract phrasing: The abstract's presentation is saturated with overly complex or ambiguous terms. It needs to be simplified to clearly convey the core contribution.

  • Overuse of Alphabet Soup: The paper relies heavily on excessive abbreviations and acronyms (e.g., SPPV, SSTOA), which significantly hurts readability and clutters the text.

  • Potential error in Figure 1: The ground-truth (GT) masks shown for the "plausible" and "anomalous" edits appear to be identical. Is this a presentation error? Please clarify.

  • Unclear labeling in Figure 3 (c): What exactly does the target group labeled "artifacts" represent in this context? A clearer definition or explanation is needed.

7 comments
AuthorsOpenReview ↗

Response to Reviewer a8dv— Part 1 of 5: Stylized Images, Localization Metrics, and Edit Size

Response to Weakness #1.1

  1. Stylized-image construction. We randomly sampled 1,000 images from norod78/cartoon-blip-captions and ran the same prompt-generation, editing, mask-generation, and overlap-filtering pipeline. Qwen-Image-Edit generated the corresponding 3,956 edited images; 1,341 pairs passed the pair-mask IoU ≥0.5 filter.

  2. Localization results. We evaluated all seven localizers on these 1,341 pairs. Table R1 shows higher IoU on anomalous edits for every method, with relative gaps from 0.081 to 0.572. The gap therefore extends to this cartoon corpus under Qwen editing.

Table R1. Localization performance on the stylized-image dataset.

ModelΔrel\Delta_{\mathrm{rel}}
SIDA-7B0.572
ForgeryTalker0.081
ObjectFormer0.161
IML-ViT0.299
CAT-Net0.164
PSCC-Net0.251
TruFor0.281

Response to Weakness #1.2

1. Purpose of the relative IoU metric. For each edited condition z∈pl,anz\in\\{\mathrm{pl},\mathrm{an}\\}, we compute Jiz=IoU⁡(M^iz,Miz)J_i^z=\operatorname{IoU}(\hat M_i^z,M_i^z) against that output's own edit mask. We then measure the performance difference as Δrel=(Jˉan−Jˉpl)/Jˉan\Delta_{\mathrm{rel}}=(\bar J_{\mathrm{an}}-\bar J_{\mathrm{pl}})/\bar J_{\mathrm{an}}, where a positive value indicates lower localization IoU on plausible edits. This paired metric compares the two edited conditions and it does not evaluate an unedited image, which has no manipulated region to localize.

2. False-positive rates on unedited images. With an empty ground-truth manipulation mask, any pixel predicted as manipulated is a false positive. We report both measures on 500 non-edited anomalous images (Table R4 in Response to Weakness #1.4).

Response to Weakness #1.3

1. Performance vary based on edit sizes. For condition z∈pl,anz\in\\{\mathrm{pl},\mathrm{an}\\}, we define edited-region size as the fraction of image pixels covered by its binary SIGMA mask:

Aiz=∣Miz∣HiWi,A_i^z=\frac{|M_i^z|}{H_iW_i},

where ∣Miz∣|M_i^z| is the number of foreground pixels in the mask and HiWiH_iW_i is the image area. For paired localization, both members receive the same size label, determined by their mean area ratio:

Aˉi=Aipl+Aian2.\bar A_i=\frac{A_i^{\mathrm{pl}}+A_i^{\mathrm{an}}}{2}.

We define small regions by Aˉi<0.05\bar A_i<0.05, medium regions by 0.05≤Aˉi<0.200.05\le\bar A_i<0.20, and large regions by Aˉi≥0.20\bar A_i\ge0.20. We re-aggregate the official-test IoUs within each size bin, averaging within each condition and editor before taking an unweighted mean across the four editors. Tables R2(a)–R2(c) report relative gap Δrel\Delta_{\mathrm{rel}} for COCO, ImageNet, and Places. The gap is positive in all 21 small-region and all 21 medium-region model–dataset combinations, and in 18 of 21 large-region combinations, totaling 60 of 63 (Tables R2(a)–R2(c)). The semantic-plausibility gap therefore appears across region sizes for multiple models.

Table R2(a). Localization performance by edit size on COCO (small 6.0%, medium 38.1%, large 55.9%).

MethodSmall Δrel\Delta_{\mathrm{rel}}Medium Δrel\Delta_{\mathrm{rel}}Large Δrel\Delta_{\mathrm{rel}}
SIDA-7B0.7610.1680.487
ForgeryTalker0.3270.2250.225
ObjectFormer0.4500.2160.086
IML-ViT0.2770.2370.046
CAT-Net0.1320.011−0.317
PSCC-Net0.1440.0710.041
TruFor0.4530.2220.253

Table R2(b). Localization performance by edit size on ImageNet (small 2.7%, medium 25.0%, large 72.2%).

MethodSmall Δrel\Delta_{\mathrm{rel}}Medium Δrel\Delta_{\mathrm{rel}}Large Δrel\Delta_{\mathrm{rel}}
SIDA-7B0.5620.3790.332
ForgeryTalker0.1770.2510.179
ObjectFormer0.1990.1370.094
IML-ViT0.1290.1370.110
CAT-Net0.2430.220−0.030
PSCC-Net0.2840.066−0.009
TruFor0.0660.2370.123

Table R2(c). Localization performance by edit size on Places (small 6.2%, medium 36.0%, large 57.7%).

MethodSmall Δrel\Delta_{\mathrm{rel}}Medium Δrel\Delta_{\mathrm{rel}}Large Δrel\Delta_{\mathrm{rel}}
SIDA-7B0.3770.4750.386
ForgeryTalker0.1960.2540.236
ObjectFormer0.2250.2480.295
IML-ViT0.1110.3030.285
CAT-Net0.2340.4170.287
PSCC-Net0.1740.2630.262
TruFor0.2110.3270.386
AuthorsOpenReview ↗

Response to Reviewer a8dv— Part 2 of 5: Edit Characteristics, Non-edited Images, and Experimental Setup

Response to Weakness #1.3

2. Foreground/background location. The dataset does not provide strict foreground/background annotations, so we define a proxy using the prompt and edit-mask position. An unambiguous foreground or background reference in the prompt determines the label. Otherwise, we assign the foreground label when the plausible-mask centroid lies within the central 50% of both image dimensions (25%–75%), and the background label otherwise. Tables R3(a)–R3(c) show higher anomalous-edit IoU in 19 of 21 foreground-proxy and all 21 background-proxy model–dataset combinations. Thus, the semantic-plausibility gap appears in both location groups and is generally more pronounced in the background group.

Table R3(a). Localization performance by foreground/background proxy on COCO (foreground 80.9%, background 19.1%).

MethodForeground Δrel\Delta_{\mathrm{rel}}Background Δrel\Delta_{\mathrm{rel}}
SIDA-7B0.3230.652
ForgeryTalker0.2090.359
ObjectFormer0.0920.383
IML-ViT0.0790.371
CAT-Net−0.1790.204
PSCC-Net−0.0030.312
TruFor0.2030.367

Table R3(b). Localization performance by foreground/background proxy on ImageNet (foreground 89.8%, background 10.2%).

MethodForeground Δrel\Delta_{\mathrm{rel}}Background Δrel\Delta_{\mathrm{rel}}
SIDA-7B0.3300.541
ForgeryTalker0.1870.156
ObjectFormer0.0930.250
IML-ViT0.0960.262
CAT-Net0.0460.333
PSCC-Net0.0060.226
TruFor0.1450.282

Table R3(c). Localization performance by foreground/background proxy on Places (foreground 78.0%, background 22.0%).

MethodForeground Δrel\Delta_{\mathrm{rel}}Background Δrel\Delta_{\mathrm{rel}}
SIDA-7B0.4060.519
ForgeryTalker0.2340.253
ObjectFormer0.2450.377
IML-ViT0.2520.326
CAT-Net0.2960.461
PSCC-Net0.2350.341
TruFor0.3150.357

3. The results of different edit actions (e.g., removal, insertion, style modification). We use replacement for both plausible and anomalous edits to hold the edit operation fixed and focus the analysis on semantic plausibility. Extending the paired evaluation to these operations is a direction for future work.

Response to Weakness #1.4

1. Dataset construction. We collected candidate photographs from public photography communities through keyword searches and photography groups, focusing on unusual apparent scales or interactions created by perspective, occlusion, and camera viewpoint. We manually inspected the images and their source pages, excluded weak anomalous relationships and obvious AI-generated or composited images, and removed exact and near duplicates using SHA-256 and perceptual hashes. Representative images and their localization visualizations will be added to the paper's appendix.

2. Localization performance. Since these images have no edited region, we measure false-positive rates. Pixel false-positive rate is the fraction of output-grid pixels exceeding threshold τ\tau, averaged equally over images; image false-positive rate is the proportion of images with at least one such pixel. Table R4 reports all nine localizers. At τ=0.5\tau=0.5, mean false-positive pixel area ranges from 0.37% for SIDA-7B to 11.99% for ObjectFormer, with 11.52% for FMAE and 11.17% for PSCC-Net.

These results show that localization models can flag non-edited anomalous scenes as manipulated, including at stricter prediction thresholds.

Table R4. False positives on 500 non-edited anomalous images.

LocalizerPixel FP, 0.5Pixel FP, 0.9Image FP, 0.5
SIDA-7B0.37%0.11%1.8%
ForgeryTalker6.69%0.70%100.0%
ObjectFormer11.99%10.00%94.0%
IML-ViT3.96%1.87%100.0%
SparseViT1.49%0.33%63.2%
FMAE11.52%3.17%99.6%
CAT-Net1.85%0.63%61.4%
PSCC-Net11.17%3.11%97.0%
TruFor1.77%0.89%46.6%

Joint response to Weakness #2.1 and Additional Question #1

We use Chinese and English to broaden instruction coverage and provide two traceable language realizations of the same intended semantic contrast. Each plausible–anomalous comparison is performed within the same language, keeping prompt language fixed when measuring the localization gap. Tables 4–6 pool both languages and quantify the plausible–anomalous gap across the bilingual collection. A separate language-stratified evaluation would quantify the relationship between prompt language and the gap, with controlled encoder analyses needed to attribute any language-specific differences to the text encoder.

AuthorsOpenReview ↗

Response to Reviewer a8dv— Part 3 of 5: Closed-source Evaluation, Prompt Controls, and Violation Types

Response to Weakness #2.2

We selected Qwen-Image-Edit-2511, FireRed-Image-Edit-1.1, LongCat-Image-Edit, and FLUX.2-dev because they provide public, reproducible checkpoints, support instruction-guided replacement, and can follow the same paired prompts under controlled generation settings. Their architectures also vary in backbone scale and multimodal processing: Qwen is a double-stream DiT; FireRed adds 3D RoPE; LongCat is a 6B hybrid; FLUX.2-dev is a 32B rectified-flow model.

Response to Weakness #2.3

Thank you for suggesting an evaluation of a closed-source localizer. We added TextIn, a commercial image-manipulation detection service from INTSIG (合合信息), and compared it with all nine public checkpoints on the same randomly sampled 500-pair subset.

Evaluate closed-source model. The subset contains 167 COCO, 167 ImageNet, and 166 Places pairs, covering the four editors and six violation types. We convert TextIn's returned PS-tamper regions and confidence scores into localization maps, then evaluate them at threshold 0.5 against each edited image's SIGMA mask. As shown in Table R5, TextIn obtains mean IoUs of 0.077 on plausible edits and 0.129 on anomalous edits, with a relative gap of 0.403. All ten localizers have positive gaps on this subset.

Table R5. Closed- and open-source localization on the same 500-pair audit subset.

LocalizerΔrel\Delta_{\mathrm{rel}}
SIDA-7B0.750
ForgeryTalker0.366
ObjectFormer0.250
IML-ViT0.205
SparseViT0.203
FMAE0.089
CAT-Net0.197
PSCC-Net0.115
TruFor0.357
TextIn0.403

Response to Weakness #3.1

All four editors receive exactly the same prompts. For each source image and language, we generate one plausible–anomalous instruction pair and apply it unchanged to Qwen, FireRed, LongCat, and FLUX.2. Within each editor, the two conditions use the same generation settings. The final release aligns 6,500 source photographs across all four editors. Differences in retained-pair counts arise from editor-specific mask-overlap filtering and the retention of Chinese and English variants, while the input prompts remain identical across editors.

Response to Weakness #3.2

We characterize abnormality through six semantic-violation types: C1 physical attribute, C2 function or affordance, C3 biological structure, C4 spatial relation, C5 interaction relation, and C6 scene context. Tables R6(a)–R6(c) now report relative gap Δrel\Delta_{\mathrm{rel}} separately for each violation type and source dataset, averaged equally across the four editors. The gap is positive in 37/42 COCO, 38/42 ImageNet, and 42/42 Places model–category comparisons, totaling 117/126. These results show that lower localization performance on plausible edits occurs across all six violation types, supporting a semantic-plausibility shortcut shared by multiple localizers across different forms of semantic conflict.

Table R6(a). Localization performance by violation type on COCO.

MethodC1 Δrel\Delta_{\mathrm{rel}}C2 Δrel\Delta_{\mathrm{rel}}C3 Δrel\Delta_{\mathrm{rel}}C4 Δrel\Delta_{\mathrm{rel}}C5 Δrel\Delta_{\mathrm{rel}}C6 Δrel\Delta_{\mathrm{rel}}
SIDA-7B0.1170.8670.5350.4150.405−0.145
ForgeryTalker0.2760.1620.2280.2130.1530.210
ObjectFormer0.1640.1300.1150.0240.2160.282
IML-ViT0.1350.3160.1450.0170.326−0.039
CAT-Net−1.0410.305−0.2600.2200.0720.125
PSCC-Net0.0080.1550.1070.114−0.3610.157
TruFor0.2720.1950.2340.6430.0330.269

Table R6(b). Localization performance by violation type on ImageNet.

MethodC1 Δrel\Delta_{\mathrm{rel}}C2 Δrel\Delta_{\mathrm{rel}}C3 Δrel\Delta_{\mathrm{rel}}C4 Δrel\Delta_{\mathrm{rel}}C5 Δrel\Delta_{\mathrm{rel}}C6 Δrel\Delta_{\mathrm{rel}}
SIDA-7B−0.0250.3350.4980.3940.3520.414
ForgeryTalker0.1150.1420.1060.2950.2300.238
ObjectFormer0.0600.1380.0210.140−0.0510.327
IML-ViT0.0980.1000.0270.2060.0880.214
CAT-Net0.0550.0740.0120.1880.1300.151
PSCC-Net−0.052−0.0350.0180.0970.0410.117
TruFor0.1730.1200.0640.3290.1300.205

Table R6(c). Localization performance by violation type on Places.

MethodC1 Δrel\Delta_{\mathrm{rel}}C2 Δrel\Delta_{\mathrm{rel}}C3 Δrel\Delta_{\mathrm{rel}}C4 Δrel\Delta_{\mathrm{rel}}C5 Δrel\Delta_{\mathrm{rel}}C6 Δrel\Delta_{\mathrm{rel}}
SIDA-7B0.1760.6480.5750.2380.3630.497
ForgeryTalker0.1600.2410.1750.3630.2880.181
ObjectFormer0.1000.3210.3220.2920.2010.431
IML-ViT0.0770.3540.2780.3850.3070.287
CAT-Net0.0840.3240.4190.4930.4800.257
PSCC-Net0.0550.2940.3000.3600.2900.275
TruFor0.2330.3500.3610.3460.3860.314
Reviewer_kYroOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper examines a semantic-plausibility shortcut; image forgery localizers identify edits whose content is anomalous but may miss forged regions that look plausible. The authors introduce PAPE, a benchmark of triplets containing the original image, plausible and anomalous forgeries. PAPE spans four image editing models, three datasets, and six anomaly categories.The evaluation reports IoU for both edit types, over the seven tested image forgery localization models, plausible edits produce lower IoU score.

Critical strengths
  1. The evaluation design is rigorous, matching the region and editing settings is more informative than previous investigations into semantic shortcut learning based on differing images with held-out semantic forgery concepts. PAPE specifically tests sensitivity in semantic plausibility with matched edits, making this concern directly testable. This increases my confidence that we are measuring this particular phenomenon.
  2. The noted gap appears across the seven localization models, four editors, and three domains, this supports a general and consistent observation. The significance of the finding is substantiated by this generalization rather than it being an isolated failure mode for a specific model or dataset.
  3. The benchmark is a valuable and novel addition to existing benchmarks to test for the specific semantic shortcut learning failure mode. This strengthens the paper's contribution through practical value.
Critical weaknesses
  1. The differing mask sizes for plausible and anomalous edits confound the evaluation. Further, the distribution of mask sizes for both types, not just the average mask size is of importance to the IoU scores. The IoU is sensitive to mask size; the same absolute error on smaller regions suffer larger IoU penalties. Stratification, the distribution of mask sizes, and the F1 score are essential to analyzing the effect of this bias. It remains unclear how much of the gap reflects region size rather than semantic sensitivity.
  2. The semantic contrast remains unvalidated by human assessment. The main contribution relies on the semantic contrast being accurate, human validation would increase my confidence in the methodology.
  3. The paper would benefit from more structured writing and figure/table organization. Throughout the paper, there is excessive white space and the figures and tables in Section 6 are ill-placed. For example, the numbers in Figure 1 overwrite the model name and are too small to read well. The appendix likewise contains excessive white space. Polishing the structure, tables, and figures would greatly improve the paper's readability and my assessment.
Questions for authors
  1. Could the authors expand on weakness (1) and provide more metrics about the mask size effect on the gap and the mask size distributions?
  2. Could the authors provide a human assessment of a subset of the data?
  3. The remaining points are the structure and organization of the paper; if the authors provide an improved version and address the above questions, my assessment would be positively influenced.
8 comments
AuthorsOpenReview ↗

Response to Reviewer kYro—Part 1 of 3: Mask-Size Distributions and Stratified Localization

Joint response to Weakness #1 and Question #1

Thank you for highlighting region size as a potential confound and for requesting distributions, size-stratified results, and F1 scores. We address these points through mask-area statistics, paired localization within size bins, and pixel-level localization F1. The two conditions have substantially overlapping area distributions, and the localization gap persists within size strata and under F1, supporting the plausibility-related localization difference beyond overall mask-size variation. The detailed responses are organized below.

1. Edited-region size and its distributions. For condition z∈pl,anz\in\\{\mathrm{pl},\mathrm{an}\\}, we define edited-region size as the fraction of image pixels covered by its binary SIGMA mask:

Aiz=∣Miz∣HiWi,A_i^z=\frac{|M_i^z|}{H_iW_i},

where ∣Miz∣|M_i^z| is the number of foreground pixels in the mask and HiWiH_iW_i is the image area. We compute the area statistics on all 47,747 retained pairs in the release, matching Table 3 of the paper. Table R1 reports condition-wise means, medians, interquartile ranges, and size-bin proportions for each source dataset. We define small regions by A<0.05A<0.05, medium regions by 0.05≤A<0.200.05\le A<0.20, and large regions by A≥0.20A\ge0.20.

As shown in Table R1, plausible and anomalous masks have means of 0.318 and 0.339, medians of 0.254 and 0.276, and overlapping interquartile ranges of 0.131–0.445 and 0.141–0.477. Anomalous masks are slightly larger on average, with 2.7–3.5 percentage points more large masks across the three datasets. Paired area similarity, min⁡(Aipl,Aian)/max⁡(Aipl,Aian)\min(A_i^{\mathrm{pl}},A_i^{\mathrm{an}})/\max(A_i^{\mathrm{pl}},A_i^{\mathrm{an}}), has mean 0.837 and median 0.868.

Table R1. Mask-area statistics and distributions by source dataset and condition.

SourceMaskMeanMedianQ1–Q3Small (<5<5\\%)Medium (5–20%)Large (≥20\ge20\\%)
COCOPlausible0.2670.2060.107–0.3678.7%40.0%51.3%
COCOAnomalous0.2850.2220.116–0.3958.0%37.2%54.8%
ImageNetPlausible0.3740.3110.175–0.5223.0%26.7%70.3%
ImageNetAnomalous0.4000.3440.188–0.5592.7%24.3%73.0%
PlacesPlausible0.3000.2300.116–0.4257.0%37.2%55.8%
PlacesAnomalous0.3150.2510.126–0.4485.9%35.5%58.6%
AllPlausible0.3180.2540.131–0.4456.0%34.0%60.0%
AllAnomalous0.3390.2760.141–0.4775.3%31.7%63.0%

2. Localization within edited-region size bins. For paired localization, both members receive the same size label, determined by their mean area ratio:

Aˉi=Aipl+Aian2.\bar A_i=\frac{A_i^{\mathrm{pl}}+A_i^{\mathrm{an}}}{2}.

We apply the same small, medium, and large thresholds to Aˉi\bar A_i. Under this paired definition, 5.5%, 32.5%, and 62.1% of retained pairs fall into the three bins, respectively. We then re-aggregate the official-test IoUs within each size bin, averaging within each condition and editor before taking an unweighted mean across the four editors. Tables R2(a)–R2(c) report plausible IoU JplJ_{\mathrm{pl}}, anomalous IoU JanJ_{\mathrm{an}}, and relative gap Δrel\Delta_{\mathrm{rel}} for COCO, ImageNet, and Places.

The gap is positive in all 21 small-region and all 21 medium-region model–dataset combinations, and in 18 of 21 large-region combinations, totaling 60 of 63 (Tables R2(a)–R2(c) in commnet 2). The large-region exceptions are CAT-Net on COCO and ImageNet, and PSCC-Net on ImageNet, where the two IoUs are nearly equal (0.151 versus 0.150). The localization gap therefore appears across region sizes for multiple models.

Table R2(a). Localization performance by mask size on COCO (small 6.0%, medium 38.1%, large 55.9%).

MethodSmall JplJ_{\mathrm{pl}}Small JanJ_{\mathrm{an}}Small Δrel\Delta_{\mathrm{rel}}Medium JplJ_{\mathrm{pl}}Medium JanJ_{\mathrm{an}}Medium Δrel\Delta_{\mathrm{rel}}Large JplJ_{\mathrm{pl}}Large JanJ_{\mathrm{an}}Large Δrel\Delta_{\mathrm{rel}}
SIDA-7B0.0040.0150.7610.0320.0390.1680.0270.0520.487
ForgeryTalker0.0380.0560.3270.0870.1120.2250.1670.2160.225
ObjectFormer0.1020.1860.4500.1740.2210.2160.2130.2330.086
IML-ViT0.1840.2550.2770.1620.2130.2370.1280.1340.046
CAT-Net0.1390.1600.1320.0630.0630.0110.0410.031−0.317
PSCC-Net0.1440.1680.1440.1380.1480.0710.1400.1460.041
TruFor0.1190.2180.4530.0950.1220.2220.0450.0610.253
AuthorsOpenReview ↗

Response to Reviewer kYro—Part 2 of 3: Size-Stratified Results and Localization F1

Joint response to Weakness #1 and Question #1

Table R2(b). Localization performance by mask size on ImageNet (small 2.7%, medium 25.0%, large 72.2%).

MethodSmall JplJ_{\mathrm{pl}}Small JanJ_{\mathrm{an}}Small Δrel\Delta_{\mathrm{rel}}Medium JplJ_{\mathrm{pl}}Medium JanJ_{\mathrm{an}}Medium Δrel\Delta_{\mathrm{rel}}Large JplJ_{\mathrm{pl}}Large JanJ_{\mathrm{an}}Large Δrel\Delta_{\mathrm{rel}}
SIDA-7B0.0100.0240.5620.0230.0380.3790.0220.0330.332
ForgeryTalker0.0450.0550.1770.1070.1430.2510.2270.2770.179
ObjectFormer0.1150.1440.1990.1930.2240.1370.1710.1890.094
IML-ViT0.2860.3280.1290.2110.2450.1370.1320.1480.110
CAT-Net0.1470.1940.2430.0850.1090.2200.0470.046−0.030
PSCC-Net0.1500.2090.2840.1870.2010.0660.1510.150−0.009
TruFor0.2330.2490.0660.1300.1710.2370.0560.0640.123

Table R2(c). Localization performance by mask size on Places (small 6.2%, medium 36.0%, large 57.7%).

MethodSmall JplJ_{\mathrm{pl}}Small JanJ_{\mathrm{an}}Small Δrel\Delta_{\mathrm{rel}}Medium JplJ_{\mathrm{pl}}Medium JanJ_{\mathrm{an}}Medium Δrel\Delta_{\mathrm{rel}}Large JplJ_{\mathrm{pl}}Large JanJ_{\mathrm{an}}Large Δrel\Delta_{\mathrm{rel}}
SIDA-7B0.0290.0470.3770.0310.0590.4750.0230.0370.386
ForgeryTalker0.0310.0390.1960.0860.1160.2540.1730.2270.236
ObjectFormer0.1240.1600.2250.1630.2170.2480.1370.1950.295
IML-ViT0.2750.3090.1110.1720.2460.3030.0860.1210.285
CAT-Net0.1620.2120.2340.0630.1080.4170.0380.0530.287
PSCC-Net0.1540.1870.1740.1260.1720.2630.0970.1320.262
TruFor0.2450.3100.2110.1490.2220.3270.0580.0940.386

3. Localization measured with F1. We also evaluate pixel-level F1 between each predicted binary mask M^iz\hat M_i^z and its condition-specific SIGMA mask MizM_i^z:

F1iz=2∣M^iz∩Miz∣∣M^iz∣+∣Miz∣.F1_i^z=\frac{2|\hat M_i^z\cap M_i^z|}{|\hat M_i^z|+|M_i^z|}.

We follow the aggregation protocol of Tables 4–6 in the paper: average within each condition and editor, then equally across the four editors. The relative F1 gap is ΔrelF1=(Fˉ1an−Fˉ1pl)/Fˉ1an\Delta_{\mathrm{rel}}^{F1}=(\bar F1_{\mathrm{an}}-\bar F1_{\mathrm{pl}})/\bar F1_{\mathrm{an}}. Tables R3(a)–R3(c) show higher F1 on anomalous edits in 20 of 21 model–dataset combinations. The exception is CAT-Net on COCO, with plausible and anomalous F1 of 0.072 and 0.070. Together with the within-size comparisons, these results show that the gap persists across region-size strata and with both IoU and F1.

Table R3(a). Localization F1 on COCO.

ModelFˉ1pl\bar F1_{\mathrm{pl}}Fˉ1an\bar F1_{\mathrm{an}}ΔrelF1\Delta_{\mathrm{rel}}^{F1}
SIDA-7B0.0360.0590.382
ForgeryTalker0.2020.2530.202
ObjectFormer0.2460.2910.156
IML-ViT0.1920.2310.168
CAT-Net0.0720.070−0.027
PSCC-Net0.1930.2070.069
TruFor0.0870.1190.267

Table R3(b). Localization F1 on ImageNet.

ModelFˉ1pl\bar F1_{\mathrm{pl}}Fˉ1an\bar F1_{\mathrm{an}}ΔrelF1\Delta_{\mathrm{rel}}^{F1}
SIDA-7B0.0270.0420.360
ForgeryTalker0.2820.3410.174
ObjectFormer0.2320.2620.114
IML-ViT0.2190.2460.111
CAT-Net0.0830.0920.102
PSCC-Net0.2230.2300.032
TruFor0.1000.1210.179

Table R3(c). Localization F1 on Places.

ModelFˉ1pl\bar F1_{\mathrm{pl}}Fˉ1an\bar F1_{\mathrm{an}}ΔrelF1\Delta_{\mathrm{rel}}^{F1}
SIDA-7B0.0330.0580.436
ForgeryTalker0.2080.2660.217
ObjectFormer0.1940.2650.267
IML-ViT0.1780.2400.260
CAT-Net0.0730.1120.351
PSCC-Net0.1580.2090.247
TruFor0.1270.1900.331
AuthorsOpenReview ↗

Response to Reviewer kYro— Part 3 of 3: Human Verification and Presentation Revisions

Joint response to Weakness #2 and Question #2

Thank you for requesting human assessment of the semantic contrast. We completed an audit of 500 randomly sampled English pairs, assessing both SIGMA masks and whether the anomalous image realizes its assigned violation. Mask quality is similar across the two conditions, while 74.4% of anomalous outputs visibly realize the intended violation. Table R4 reports the results and the assessment and findings are detailed below.

1. Human agreement with the SIGMA mask. We randomly sampled 500 English pairs: 167 from COCO, 167 from ImageNet, and 166 from Places, covering the six violation types and four editors. Three human experts assessed mask quality on a 1–5 scale, where 1 indicates that the mask misses the edited region and 5 indicates that it matches the visible change. They also recorded whether the anomalous image realizes its assigned violation and whether the plausible edit is visually almost unchanged from the source. As shown in Table R4, plausible and anomalous masks receive mean scores of 4.586 and 4.576, respectively, with 90.6% and 91.0% rated 4 or 5. Within pairs, ratings are equal in 80.6% of cases, higher for the plausible mask in 10.4%, and higher for the anomalous mask in 9.0%. These high scores indicate close correspondence between SIGMA masks and human judgments of the visible edited regions, while the paired ratings show similar mask quality across conditions.

2. Anomalous image actually instantiates the sampled violation. Human inspection finds that 372/500 anomalous images (74.4%) realize their assigned violation (Table R4). This shows that most anomalous outputs retained by our automated curation pipeline visibly instantiate the intended semantic conflict.

Table R4. Human visual assessment of 500 English pairs (Pl: plausible; An: anomalous).

SourcePairsMean mask score, 1–5 (Pl / An)Masks rated ≥4 (Pl / An)Assigned violation realizedPlausible near-copy
COCO1674.898 / 4.85698.2% / 96.4%65.3% (109/167)0.0% (0/167)
ImageNet1674.623 / 4.58794.0% / 93.4%82.6% (138/167)0.0% (0/167)
Places1664.235 / 4.28379.5% / 83.1%75.3% (125/166)0.0% (0/166)
All5004.586 / 4.57690.6% / 91.0%74.4% (372/500)0.0% (0/500)

Joint response to Weakness #3 and Question #3

Thank you for the suggestions on structure, layout, and figure readability. We have prepared a redrawn Figure 1 and provided the requested additional analyses and human assessment in the responses above. The revised manuscript will incorporate these results alongside the presentation changes detailed below.

1. Writing and experimental structure. We will separate the evaluation protocol from the results and organize the discussion around the paired localization comparison, region-size controls, and human validation. Each results paragraph will state its main finding and explain the supporting figure or table, making the connection between the experimental question and the evidence explicit.

2. Figure placement and page layout. We will place the figures and tables in Section 6 beside the paragraphs that introduce and interpret them, reduce excess vertical spacing, and consolidate short text blocks that leave large gaps. We will apply the same layout revisions to the appendix so that related results and explanations appear together.

3. Figure 1 readability. The redrawn figure separates model names, relative-gap bars, and absolute IoU values into distinct columns. It uses larger labels, readable prompt boxes, and subfigure captions below the panels. These changes remove the overlap between numerical values and model names and make the paired examples and localization results easier to follow. We will use this version in the revised manuscript.

4. Incorporation of the additional evidence. The responses above now include mask-area distributions (Table R1), localization within size bins (Tables R2(a)–R2(c)), pixel-level F1 (Tables R3(a)–R3(c)), and the completed 500-pair human assessment (Table R4). We will incorporate these results with their corresponding interpretations into the revised paper, connecting the region-size and annotation-quality checks to the main localization findings.

Reviewer_4qFWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that a per-image localization score cannot tell forensic evidence from a semantic-plausibility shortcut: a model may localize an anomalous replacement of a target and miss a plausible replacement of the same target. PAPE is a test-only benchmark whose unit is a triplet. One source photograph, one editor, and shared settings produce a plausible edit and an anomalous edit. Both edited images are labeled manipulated. Masks are estimated separately for each output by SIGMA. A pair is kept only if the two masks have IoU at least 0.5. The release has 6,500 photographs (2,000 COCO, 2,500 ImageNet, 2,000 Places) and 47,747 language-specific triplets from four editors (Qwen-Image-Edit, FireRed-Image-Edit, LongCat-Image-Edit, Flux.2-dev), giving 95,494 edited images. Prompts come from Qwen3.5-9B. A rule checker accepts a prompt if required fields are non-empty and both languages contain the word "replace"; it does not judge whether the image realizes the intended violation. Mean retained mask IoU is 0.770. On a stratified sample of 1,920 pairs, anomalous masks have mean foreground fraction 0.349 versus 0.324 for plausible masks. Seven public localizers are scored one image at a time at a fixed threshold of 0.5. The reported quantity is the relative IoU gap, (mean anomalous IoU minus mean plausible IoU) divided by mean anomalous IoU. Editor-averaged gaps are positive for every method on ImageNet and Places and for six of seven on COCO. The positive averages run from 0.025 (PSCC-Net, ImageNet) to 0.422 (SIDA-7B, Places). CAT-Net on COCO is negative, at -0.059. No new localizer is trained.

Critical strengths

First, the comparison unit is actually matched and large enough to audit. All 47,747 retained pairs have mask IoU at least 0.5, the mean is 0.770, and 67.6% are at or above 0.7. Both members are labeled as manipulations; only the intended anomaly bit changes. That bookkeeping is the right one for this question, and the images are scored without the source and without the pair, which matches source-free localization. Second, the sign is not a one-model anecdote, and the counterexamples are in the paper. CAT-Net's COCO average is -0.059. Flux.2 flips IML-ViT and CAT-Net on COCO (-0.084 and -0.121). Section 6 says low IoU makes CAT-Net's Places ratio of 0.333 unstable, and that generative traces can contribute. Publishing that heterogeneity is a strength of the empirical section.

Critical weaknesses

First, intended plausibility is confounded with how much the image changes. The motivating example replaces a cargo cage with another cage or with a large cat, which changes identity, color, texture, and scale. Mask IoU at least 0.5 does not equalize pixel change, boundary length, or frequency residual. Edit-success rate is not reported. Until those distributions are shown side by side, a positive relative gap does not establish a semantic-plausibility shortcut. This is the weakness that supports rejection. Second, the ground-truth mask is an unvalidated semantic segmenter. SIGMA is run on each output and its instruction. Anomalous instructions have a larger intended semantic gap, so mask completeness can differ by condition, and a multimodal localizer that shares that sensitivity would look better on the anomalous member. "Annotated" here is automatic. The taxonomy that justifies the benchmark (six violation types) is never a column in Tables 4-6.

Questions for authors

Clarification: On a random sample of retained pairs, what fraction of plausible edits are near-copies of the source, and what are the paired distributions of pixel change or LPIPS for the two members? If the plausible member is systematically a smaller edit, my reading of Tables 4-6 changes. Clarification: What is human agreement with the SIGMA mask, and with the claim that the anomalous image actually instantiates the sampled violation, on a few hundred pairs? SPPV checks the string "replace" and the category id, not the image. Clarification: Does the gap remain if both members are scored against a source-output difference map, with the same 0.5 threshold, rather than against SIGMA? These are clarification requests about the existing release. I am not asking for a new localizer.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 4qFW—Part 1 of 3: Pixel Change and Edit-Magnitude Controls

Joint response to Clarification #1 and Weakness #1

Thank you for highlighting edit magnitude as a potential confound and for requesting paired statistics and edit-success rates.

1. Fraction of near-copy edits. We define pixel change as the mean absolute RGB difference between an edited image and its source inside the edited image's SIGMA mask. With RGB values normalized to [0,1][0,1],

\frac{1}{3|M_i^z|} \sum_{p:M_i^z(p)=1}\sum_{c\in\\{R,G,B\\}} \left|I_i^z(p,c)-I_i^{\mathrm{real}}(p,c)\right|, \qquad z\in\\{\mathrm{pl},\mathrm{an}\\}.$$ Here, $I_i^{\mathrm{real}}$ is the source, $I_i^z$ is the edited image, and $M_i^z$ is its edit mask. We count an edit as a near-copy when $d_{\mathrm{pix}}<0.05$. We compute these statistics on all **47,747 retained pairs** in the release: 14,668 COCO, 18,488 ImageNet, and 14,591 Places pairs, matching Table 3 of the paper. We average equally across the four editors within each dataset and then across the three datasets. **Only 0.39% of plausible edits are near-copies** under this criterion, compared with 0.26% of anomalous edits (Table R2). **Table R1. Editor-averaged distributions of in-mask pixel change in 0.05-wide bins (Pl: plausible; An: anomalous; Mean: unweighted average across COCO, ImageNet, and Places).** |Pixel change|Mean Pl|Mean An| |---|---:|---:| |[0.00, 0.05)|0.39%|0.26%| |[0.05, 0.10)|3.62%|2.58%| |[0.10, 0.15)|9.85%|7.39%| |[0.15, 0.20)|16.81%|15.12%| |[0.20, 0.25)|20.38%|21.53%| |[0.25, 0.30)|17.53%|19.93%| |[0.30, 0.35)|12.16%|14.23%| |[0.35, 0.40)|7.24%|8.75%| |[0.40, 0.45)|4.58%|4.77%| |[0.45, 0.50)|3.04%|2.59%| |[0.50, 0.55)|1.78%|1.35%| |[0.55, 0.60)|1.12%|0.75%| |[0.60, 0.65)|0.66%|0.41%| |[0.65, 0.70)|0.39%|0.20%| |[0.70, 0.75)|0.27%|0.09%| |[0.75, 0.80)|0.11%|0.02%| |[0.80, 0.85)|0.04%|0.02%| |[0.85, 0.90)|0.02%|0.01%| |[0.90, 0.95)|0.00%|0.00%| |[0.95, 1.00]|0.00%|0.00%| **Table R2. Sensitivity of low-change rates to the pixel-change threshold.** |Source|Pl <0.02|An <0.02|Pl <0.05|An <0.05|Pl <0.1|An <0.1| |---|---:|---:|---:|---:|---:|---:| |Mean across datasets|0.02%|<0.01%|0.39%|0.26%|4.01%|2.84%| **2. Paired distributions of pixel change.** For each retained pair, we compare the plausible and anomalous pixel changes directly. The resulting proportions are **52.92% for plausible < anomalous** and **47.08% for plausible ≥ anomalous**, both close to 50%. The two conditions have similar means (0.2651 versus 0.2690), medians (0.2474 versus 0.2574), and strongly overlapping interquartile ranges. The mean paired difference is +0.0039 (95% CI: +0.0015 to +0.0063), indicating a small shift toward larger anomalous edits. Confidence intervals use random resampling of source-image groups within each dataset, keeping the editor and language variants together. **The distributions show substantial overlap rather than a separation into small plausible edits and large anomalous edits.** **3. Localization after controlling pixel-change magnitude.** To test whether edit magnitude accounts for the localization gaps in Tables 4–6 of the paper, we define the within-pair pixel-change similarity as $$s_{\mathrm{pix}}(i)= \frac{\min\\!\left(d_{\mathrm{pix}}(i,\mathrm{pl}),\\,d_{\mathrm{pix}}(i,\mathrm{an})\right)} {\max\\!\left(d_{\mathrm{pix}}(i,\mathrm{pl}),\\,d_{\mathrm{pix}}(i,\mathrm{an})\right)}.$$ We first remove pairs in which either member has pixel change below **0.05**, retaining 99.5% of official-test pairs. Among these remaining pairs, we require $s_{\mathrm{pix}}\ge0.8$ or $s_{\mathrm{pix}}\ge0.9$. These subsets retain 51.3% and 28.4% of the original official-test pairs, respectively (Table R5). Under the stricter requirement, mean pixel changes are 0.2585 for plausible edits and 0.2591 for anomalous edits, while the mean absolute paired difference decreases from 0.0782 to 0.0127. Under the stricter 0.9 requirement, it remains positive in 19 of 21 combinations (90.5%); the exceptions are CAT-Net and PSCC-Net on COCO (Tables R6(a)–R6(c) in next comment). The persistence of this gap when the two members have closely matched pixel-change magnitudes supports **the semantic-plausibility shortcut beyond the measured edit-magnitude difference.** **Table R5. Pixel-change balance in the common official-test set and its nested control subsets.** |Subset|Retained|Mean plausible change|Mean anomalous change|Mean absolute paired difference| |---|---:|---:|---:|---:| |All official-test pairs|100.0%|0.2636|0.2688|0.0782| |Both pixel changes >=0.05|99.5%|0.2648|0.2699|0.0784| |Both >=0.05; similarity >=0.8|51.3%|0.2602|0.2614|0.0261| |Both >=0.05; similarity >=0.9|28.4%|0.2585|0.2591|0.0127|
AuthorsOpenReview ↗

Response to Reviewer 4qFW—Part 2 of 3: Localization Controls, Region Size, and Edit Success

Table R6(a). Localization after pixel-change controls on COCO, averaged across editors.

ModelPlausible IoUAnomalous IoURelative gap
SIDA-7B0.02280.0366+0.378
ForgeryTalker0.13840.1679+0.176
ObjectFormer0.17120.2035+0.158
IML-ViT0.13540.1686+0.197
CAT-Net0.05470.0530-0.033
PSCC-Net0.14630.1322-0.107
TruFor0.08220.0975+0.157

Table R6(b). Localization after pixel-change controls on ImageNet, averaged across editors.

ModelPlausible IoUAnomalous IoURelative gap
SIDA-7B0.02270.0337+0.326
ForgeryTalker0.18880.2308+0.182
ObjectFormer0.16300.1818+0.103
IML-ViT0.15230.1702+0.105
CAT-Net0.05990.0701+0.145
PSCC-Net0.16260.1662+0.021
TruFor0.08020.0970+0.174

Table R6(c). Localization after pixel-change controls on Places, averaged across editors.

ModelPlausible IoUAnomalous IoURelative gap
SIDA-7B0.02270.0443+0.487
ForgeryTalker0.13580.1745+0.222
ObjectFormer0.14270.1909+0.253
IML-ViT0.12530.1719+0.271
CAT-Net0.05280.0804+0.344
PSCC-Net0.11460.1449+0.209
TruFor0.10170.1477+0.312

4. Edited-region size and localization results. For condition z∈pl,anz\in\\{\mathrm{pl},\mathrm{an}\\}, we define edited-region size as the fraction of image pixels covered by its binary SIGMA mask:

Aiz=∣Miz∣HiWi,A_i^z=\frac{|M_i^z|}{H_iW_i},

where ∣Miz∣|M_i^z| is the number of foreground pixels in the mask and HiWiH_iW_i is the image area. We group these area ratios into small (A<0.05A<0.05), medium (0.05≤A<0.200.05\le A<0.20), and large (A≥0.20A\ge0.20) regions. Table R7 bins plausible and anomalous masks separately.

Table R7. Mask-area distribution by source dataset and condition.

SourceMaskSmall (<5<5\\%)Medium (5–20%)Large (≥20\ge20\\%)
COCOPlausible8.7%40.0%51.3%
COCOAnomalous8.0%37.2%54.8%
ImageNetPlausible3.0%26.7%70.3%
ImageNetAnomalous2.7%24.3%73.0%
PlacesPlausible7.0%37.2%55.8%
PlacesAnomalous5.9%35.5%58.6%
AllPlausible6.0%34.0%60.0%
AllAnomalous5.3%31.7%63.0%

For paired localization analysis, we assign both members a shared size label using their mean area ratio,

Aˉi=Aipl+Aian2,\bar A_i=\frac{A_i^{\mathrm{pl}}+A_i^{\mathrm{an}}}{2},

and apply the same small, medium, and large thresholds to Aˉi\bar A_i. Tables R8(a)–R8(c) report editor-averaged localization results within these bins for COCO, ImageNet, and Places. The gap is positive in all small- and medium-region model–dataset combinations and in 18 of 21 large-region combinations, totaling 60 of 63. The large-region exceptions are CAT-Net on COCO and ImageNet, and PSCC-Net on ImageNet, whose IoUs are nearly equal (0.151 versus 0.150). The semantic-plausibility gap thus appears across region sizes for multiple models.

Table R8(a). Localization performance by mask size on COCO

MethodSmall Δrel\Delta_{\mathrm{rel}}Medium Δrel\Delta_{\mathrm{rel}}Large Δrel\Delta_{\mathrm{rel}}
SIDA-7B0.7610.1680.487
ForgeryTalker0.3270.2250.225
ObjectFormer0.4500.2160.086
IML-ViT0.2770.2370.046
CAT-Net0.1320.011−0.317
PSCC-Net0.1440.0710.041
TruFor0.4530.2220.253

Table R8(b). Localization performance by mask size on ImageNet

MethodSmall Δrel\Delta_{\mathrm{rel}}Medium Δrel\Delta_{\mathrm{rel}}Large Δrel\Delta_{\mathrm{rel}}
SIDA-7B0.5620.3790.332
ForgeryTalker0.1770.2510.179
ObjectFormer0.1990.1370.094
IML-ViT0.1290.1370.110
CAT-Net0.2430.220−0.030
PSCC-Net0.2840.066−0.009
TruFor0.0660.2370.123

Table R8(c). Localization performance by mask size on Places

MethodSmall Δrel\Delta_{\mathrm{rel}}Medium Δrel\Delta_{\mathrm{rel}}Large Δrel\Delta_{\mathrm{rel}}
SIDA-7B0.3770.4750.386
ForgeryTalker0.1960.2540.236
ObjectFormer0.2250.2480.295
IML-ViT0.1110.3030.285
CAT-Net0.2340.4170.287
PSCC-Net0.1740.2630.262
TruFor0.2110.3270.386

5. Edit-success rate. Table R9 reports the mask-overlap pass rate during construction, before common-source alignment and final source selection. Among generated pairs with two usable masks, 62.7% pass the pair-mask IoU ≥0.5 filter. Subsequent alignment and source selection yield the 47,747 released pairs reported in Table 3 of the paper.

Table R9. Mask-overlap pass rates before common-source alignment (passed/generated pairs).

SourceAll
COCO58.7% (23,118/39,406)
ImageNet69.1% (27,447/39,738)
Places60.4% (23,738/39,302)
All62.7% (74,303/118,446)
AuthorsOpenReview ↗

Response to Reviewer 4qFW—Part 3 of 3: Human Verification, Violation Types, and Mask Robustness

Joint response to Clarification #2 and Weakness #2

Thank you for requesting human verification of the masks and realized violations, and for highlighting possible condition-dependent mask quality.

1. Human agreement with the SIGMA mask. We randomly sampled 500 English pairs: 167 from COCO, 167 from ImageNet, and 166 from Places, covering the six violation types and four editors. Three human experts assessed mask quality on a 1–5 scale, where 1 indicates that the mask misses the edited region and 5 indicates that it matches the visible change. As shown in Table R10, plausible and anomalous masks receive mean scores of 4.586 and 4.576, respectively, with 90.6% and 91.0% rated 4 or 5. These high scores indicate close correspondence between SIGMA masks and human judgments of the visible edited regions, while the paired ratings show similar mask quality across conditions.

2. Anomalous image actually instantiates the sampled violation. Human inspection finds that 372/500 anomalous images (74.4%) realize their assigned violation (Table R10). This shows that most anomalous outputs retained by our automated curation pipeline visibly instantiate the intended semantic conflict.

Table R10. Human visual assessment of 500 English pairs (Pl: plausible; An: anomalous).

SourcePairsMean mask score, 1–5 (Pl / An)Masks rated ≥4 (Pl / An)Assigned violation realizedPlausible near-copy
All5004.586 / 4.57690.6% / 91.0%74.4% (372/500)0.0% (0/500)

3. Localization by violation type. Table R11 reports plausible IoU JplJ_{\mathrm{pl}}, anomalous IoU JanJ_{\mathrm{an}}, and relative gap Δrel\Delta_{\mathrm{rel}} for all six intended violation types and seven localizers. The averaged gap is positive in 41 of 42 model–category combinations; the exception is CAT-Net for physical attribute violations (C1). Positive gaps occur across all six intended violation types.

Table R11. Localization performance by violation type, averaged equally across editors and source datasets.

ModelTypeJplJ_{\mathrm{pl}}JanJ_{\mathrm{an}}Δrel\Delta_{\mathrm{rel}}
SIDA-7BC10.0270.0300.107
C20.0150.0380.605
C30.0280.0620.540
C40.0310.0480.354
C50.0210.0340.375
C60.0290.0400.270
ForgeryTalkerC10.1390.1690.177
C20.1520.1850.180
C30.1540.1840.165
C40.1730.2450.291
C50.1550.2000.226
C60.1570.2000.213
ObjectFormerC10.1550.1740.107
C20.1720.2130.193
C30.1800.2150.162
C40.1800.2110.148
C50.1550.1760.120
C60.1740.2640.340
IML-ViTC10.1410.1560.101
C20.1490.2030.267
C30.1670.1980.156
C40.1440.1840.218
C50.1240.1630.238
C60.1310.1570.168
CAT-NetC10.0580.052-0.118
C20.0560.0750.250
C30.0720.0840.138
C40.0460.0700.340
C50.0480.0660.278
C60.0530.0650.183
PSCC-NetC10.1190.1190.001
C20.1460.1690.136
C30.1430.1670.147
C40.1310.1610.189
C50.1420.1470.039
C60.1440.1750.178
TruForC10.0810.1040.221
C20.0880.1150.238
C30.1120.1500.255
C40.0720.1200.400
C50.0860.1100.223
C60.0720.0980.269

Response to Clarification #3

Thank you for suggesting this alternative evaluation. Following the reviewer's suggestion, we nevertheless construct an alternative mask by thresholding the mean absolute RGB source–output difference at 0.5 and rescore all seven localizers against it, while keeping the localizer prediction threshold at 0.5. As shown in Tables R12(a)–R12(c), anomalous-edit IoU remains higher in 20 of 21 model–dataset cells: 6/7 methods on COCO and all seven methods on ImageNet and Places. The only exception is CAT-Net on COCO, where both IoUs are near the floor (0.022 versus 0.018). The observed gap is therefore not specific to SIGMA or its use of the edit instruction.

Table R12(a). Localization with source–output difference masks on COCO (difference threshold 0.5).

ModelΔrel\Delta_{\mathrm{rel}}
SIDA-7B0.511
ForgeryTalker0.192
ObjectFormer0.096
IML-ViT0.106
CAT-Net−0.228
PSCC-Net0.013
TruFor0.049

Table R12(b). Localization with source–output difference masks on ImageNet (difference threshold 0.5).

ModelΔrel\Delta_{\mathrm{rel}}
SIDA-7B0.281
ForgeryTalker0.160
ObjectFormer0.110
IML-ViT0.177
CAT-Net0.090
PSCC-Net0.085
TruFor0.158

Table R12(c). Localization with source–output difference masks on Places (difference threshold 0.5).

ModelΔrel\Delta_{\mathrm{rel}}
SIDA-7B0.371
ForgeryTalker0.210
ObjectFormer0.272
IML-ViT0.309
CAT-Net0.357
PSCC-Net0.300
TruFor0.372

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.