acceptodds
Under review as a conference paper at ICLR 2027

XPlainVerse: A Million-Scale Benchmark for Explainable Deepfake Detection

Abstract

As deepfake detection models increasingly produce natural-language explanations, their reasoning often remains weakly grounded in visual artifacts, limiting reliability and user trust. Existing benchmarks mainly evaluate classification accuracy, overlooking whether explanations reflect the actual manipulations. This gap hinders progress toward deployable, explainable deepfake detection systems. To this end, we introduce XPlainVerse, a large-scale benchmark designed for joint deepfake detection and human-centered explanation. XPlainVerse comprises one million real and manipulated images, pairing authentic images from five established sources with forgeries generated by twelve off-the-shelf image editing and synthesis models. We further propose a multi-stage filtering pipeline, Edit-Check, to verify whether manipulations satisfy their intended edits, yielding high-quality, edit-consistent examples for reasoning supervision at scale. Beyond dataset scale, XPlainVerse provides two complementary explanation styles: technical explanations for expert analysis and simplified explanations designed for non-technical users. To evaluate explanation quality beyond surface similarity, we propose novel metrics, EntityScore and EvidenceScore, that assess whether explanations capture the manipulated entities and supporting evidence reflected in the reference explanations. Human annotations on 2,000 manipulated images, together with complementary human evaluation, validate explanation and edit quality. We believe XPlainVerse will make grounded explanation quality an explicit evaluation target in deepfake detection and support scalable research on trustworthy, interpretable models.

Primary area:
datasets and benchmarks
Keywords:
Deepfakes, Explainability, VLMs

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_1hQhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

XPlainVerse is a large-scale dataset for explainable deepfake detection, containing one million images and 1.53 million image–explanation pairs. The authors combine multiple image-editing models, an edit-consistency filtering pipeline, and technical and simplified explanations. They also introduce EntityScore and EvidenceScore and evaluate detection and explanation generation under generator and perturbation shifts.

Critical strengths

1.Explainable deepfake detection is a worthwhile research direction, and the dataset provides substantial supervision across different scenes, editing models, and explanation styles.

2.Edit-Check addresses a practical problem in synthetic dataset construction: generated images do not always follow the requested edits. The human comparison of retained and discarded images provides some support for the filtering procedure.

3.The paper includes human-written references, a user study, and detailed supplementary experiments. The large gap between in-domain and out-of-domain performance also exposes weaknesses that would be missed by evaluating only familiar generators.

Critical weaknesses

1.Contribution beyond scale is not sufficiently isolated. The pipeline mainly combines existing models for editing, filtering, and annotation, while Table 1 already lists several datasets using model ensembles and quality control. This does not make the dataset unimportant, but the paper needs controlled comparisons showing what Edit-Check and dual-level explanations contribute beyond collecting more data. Appendix C also clarifies that the million images comprise 500K base images and their perturbed counterparts; the headline scale should not be interpreted as one million independent examples.

  1. Proposed metrics measure agreement with reference text, not visual grounding. EntityScore and EvidenceScore compare extracted claims between explanations without checking the image. A prediction that repeats an incorrect reference claim can score well, while a valid observation absent from the reference can be penalized. Cross-family judge agreement in Appendix I is useful, but agreement between models does not establish agreement with human judgments of factual correctness. This is a central limitation given the paper’s emphasis on grounded explanations.

  2. The annotation prompts risk producing plausible justifications rather than reliable forensic evidence. Annotators and models know that the image is manipulated and can inspect the original image; generated annotations also receive edit instructions and difference descriptions. Asking them to write “as if” they saw only the edited image does not remove this privileged information. Some prompt examples are particularly questionable: wearing a winter coat on a sunny beach is treated as evidence of manipulation, while dirt and scratches are presented as evidence of authenticity. These are neither necessary nor sufficient indicators and could teach semantic stereotypes rather than forensic reasoning.

  3. The generator-shift evaluation is strongly affected by training imbalance. Table 9 shows that 354,256 of 360,000 training fake images—approximately 98.4%—come from Gemini-2.0-Flash. The resulting OOD collapse is therefore difficult to separate from an overwhelmingly single-generator training distribution. The broader-generator experiment helps, but explicitly does not balance generators. Perturbation difficulty also increases across splits, introducing another source of shift. Generator-balanced training and separate clean/perturbed results are needed to isolate the causes of failure.

  4. Human study supports perceived quality more clearly than factual reliability or practical usefulness. The paper does include 2,000 human-written explanations and 30 evaluators, but the explanation-quality and preference stages cover only about 118–120 images each. Evaluators were recruited from institutional research groups, which limits the conclusions about ordinary users. Preference for generated explanations over human-written ones does not establish that their individual claims are correct, and reported readability does not show that explanations improve detection decisions. A blinded, claim-level audit and an actual user decision task would provide stronger evidence.

Questions for authors
  1. Can you validate EntityScore and EvidenceScore against blinded human judgments of individual claims, including incorrect reference claims and valid evidence omitted from the reference?
  2. How often do explanations contain evidence that is apparent only after seeing the original image or edit instruction? What happens when annotations are generated from the edited image alone, with an option to report insufficient visible evidence?
  3. How much of the OOD performance drop remains under generator-balanced training and identical perturbation conditions? Please report clean and perturbed results separately.
  4. At a fixed training size and generator distribution, how do label-only training, unfiltered explanations, Edit-Check explanations, and the two explanation styles compare?
  5. Are each source image, all its edited variants, and their perturbed counterparts confined to a single split? How are duplicate source images across the five source datasets handled?
  6. Do simplified explanations improve users’ detection accuracy or confidence calibration, rather than only readability and preference?
  7. Does your overview of current AIGC image detection datasets omit recent explainable benchmarks, such as Veritas's Benchmark and IvyFake? Please re-evaluate the completeness of your benchmark comparison; since recent datasets across different categories exhibit distinct styles, you should highlight the differences among these CoT' AIGC Detection datasets.

[1] Veritas: Generalizable Deepfake Detection via Pattern-Aware Reasoning [2] Ivy-fake: a unified explainable framework and benchmark for image and video aigc detection

5 comments
AuthorsOpenReview ↗

Response to Reviewer 1hQh (Part 1/3)

We are sincerely grateful to the reviewer for their careful assessment and constructive feedback. We greatly appreciate the time and effort devoted to reviewing our work.

Q1. Human validation of the metrics

We appreciate this important question. Firstly, our existing human study [Sec. 3.3; Appendix H, Stage 3, Tables 17–18] evaluates the visual grounding and completeness of reference explanations using only the edited image, without the original image, editing instructions, or difference descriptions. Across 118 images and 360 evaluations, the explanations receive 4.05/5 for grounding, 3.82/5 for completeness, and 2.40/5 for unsupported claims (lower is better). These results provide independent human evidence regarding the visual support and coverage of the reference explanations.

To further validate EntityScore and EvidenceScore against human judgments of entity and evidence correspondence, we conducted an additional human study on 100 reference–prediction explanation pairs, with two ratings per pair. Participants saw explanations labelled A/B and counted, for each explanation, the total entities and evidence observations and how many were also present in the other explanation. Using the definitions in Appendix I, we calculated scores from each response and averaged the two human scores for each pair. Table 3 compares the automatic scores with these human-derived scores.

MetricPearson rrSpearman ρ\rhoMean absolute error
EntityScore0.8630.8330.128
EvidenceScore0.8700.8360.108

Table 3: Agreement with the mean human-derived score over 100 explanation pairs. Scores lie in [0,1][0,1].

Both metrics show high correlations and low absolute errors, indicating that their scores closely reflect human assessments of the shared entities and evidence on these pairs. This supports their intended role as reference-based explanation metrics [Sec. 4; Appendix I]. We will add these details in our paper as well.

Q2. Edited-image-only explanation generation

Firstly, Our human study in Sec. 3.3 and Appendix H evaluates the original explanations using only the edited image, without the original image, editing instruction or difference description. Across 118 images and 360 evaluations, grounding receives 4.05/5 and unsupported claims receive 2.40/5, where lower is better [Stage 3, Tables 17–18]. These ratings support the visible grounding of the evaluated explanations.

To further examine what happens when annotations are generated from the edited image alone, we conducted an additional experiment with Qwen3.5-27B on 500 images, sampled approximately evenly across twelve editing generators. We withheld the original image, editing instruction and difference description used in the annotation pipeline [Sec. 3.1, Step 5], as well as the edited label. The model could either generate an explanation based on visible evidence or report insufficient visible evidence.

Under this setting, the model generated explanations for 289 images (57.8%) and reported insufficient visible evidence for 211 images (42.2%). Thus, relying on the edited image alone left a substantial proportion of the sampled images without an explanation. This motivates providing contextual guidance during annotation. The additional inputs in our pipeline are intended to help the model identify relevant regions, while the explanation prompt requires the final claims to describe evidence visible in the edited image [Appendix K.6]. Our human evaluation then assesses the resulting explanations using only the edited image [Sec. 3.3; Appendix H]. We will include this additional experiment and its prompting conditions in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer 1hQh (Part 2/3)

Q3. Generator balance and perturbations

Sec. 5 and Appendix J (Table 24) report our generator balanced training study with Qwen3.5-4B and BusterX++. The fake-image portion of this mixture covered five generators: Gemini-2.0-Flash, LongCat, Hunyuan, ICEdit, and GPT-Image-1. We excluded seven additional generators from fine-tuning and evaluated them as held-out generators: FLUX.1-dev, GPT-Image-1.5, Nano Banana 2, Nano Banana Pro, Qwen-Image-Edit, Seedream 4.5, and Wan 2.6.

To examine the role of perturbations, we evaluated both models separately on clean images and their perturbed counterparts, covering all twelve generators. Within each condition, we calculated fake-image accuracy (recall on fake images) for each generator, then averaged equally across the five training generators and, separately, the seven held-out generators.

ModelConditionSeen Fake Acc.Held-out Fake Acc.Gap
Qwen3.5-4BClean84.9%46.9%38.0 pp
Qwen3.5-4BPerturbed80.6%51.0%29.6 pp
BusterX++Clean86.1%46.5%39.6 pp
BusterX++Perturbed81.6%51.1%30.5 pp

Table 4: Clean and perturbed results after five-generator fine-tuning.

When both generator groups are evaluated on clean images, the gap remains 38.0 percentage points for Qwen and 39.6 for BusterX++. On the corresponding perturbed images, the gaps are 29.6 and 30.5 points, respectively. A substantial gap therefore persists under balanced generator training in both conditions. This supports the finding in Sec. 5 that strong performance on training generators does not ensure comparable performance on held-out generators.

Q4. Edit-Check and explanation supervision

In the construction pipeline [Sec. 3.1, Steps 3 and 5], explanations are generated for retained edits; discarded edits do not have explanations. An unfiltered-explanation training condition is therefore not available in the dataset. The human evaluation in Sec. 3.3, Fig. 4(b), and Appendix H (Stage 2) directly compares retained and discarded edits. Retained edits receive a composite edit-fidelity score of 3.77/5, compared with 2.66/5 for discarded edits. These results support Edit-Check's role in selecting higher-quality edits.

To examine the contribution of explanation supervision, we conducted an additional experiment in which we trained BusterX++ using labels alone, while preserving the same hyperparameters and training data. Table 5 compares the new label-only results with the explanation-supervised results on the same ID/OOD evaluation images. Each setting uses its intended prediction format: a class label for label-only models, and an explanation followed by the label for explanation-supervised models.

Training supervisionID F1 (%)OOD F1 (%)
Label-only88.8441.85
Label + complex explanations91.8937.14
Label + simple explanations92.4537.87

Table 5: Fake-class F1 under different supervision settings.

Compared with the explanation-supervised models, label-only training gives a similar result, with 3.05–3.61 percentage points lower ID F1 and 3.98–4.71 points higher OOD F1, thus a substantial ID/OOD gap still persists. Thus, the generalization difficulty reported in Sec. 5 is not solely a consequence of requiring models to generate explanations. It remains even when they are trained to predict labels alone. We will include this comparison and its training details in the paper.

AuthorsOpenReview ↗

Response to Reviewer 1hQh (Part 3/3)

Q5. Source splits and duplicates

Each clean image and its perturbed counterpart are kept in the same split. For the protocol in Sec. 5 and Appendix C, we chose to define ID/OOD by generator exposure. Different generators can introduce different visual artifacts, and our aim is to test whether detectors generalize beyond the generators represented during fine-tuning. We did not additionally require all edits of a source image to remain in one split. The original image and different edited versions can therefore occur across splits.

For duplicates across the five source datasets, our provenance audit compared SHA-256 file hashes and, where available, hashes of decoded image pixels, to identify exact duplicate source images.

Q6. Simplified explanations and user outcomes

Our primary objective with simplified explanations is to make the evidence behind a detection decision more accessible to non-expert users, rather than to improve their independent deepfake detection accuracy. Accordingly, our human study focuses on users' perceived understanding, memorability, and effort [Sec. 3.3; Appendix H, Stage 6, Tables 17–18]. Across 119 images and 360 evaluations, simplified explanations received 3.99/5 for immediate understanding, 3.97/5 for memorability, and 2.13/5 for effort (lower is better). These results support their perceived accessibility beyond preference alone.

Q7. Veritas and Ivy-Fake

Thank you for highlighting these benchmarks. We will include both in the dataset comparison in Table 1 and the related-work discussion in Sec. 2.

HydraFake, introduced with Veritas [1], focuses on facial forgeries and contains 100K images. Veritas provides 36K reasoning-annotated training examples that organize forensic evidence into stages such as planning and self-reflection. Ivy-Fake [2] covers both images and videos, with explanations describing spatial artifacts and, for videos, temporal inconsistencies. Its paper reports approximately 106K image/video training samples.

Table 6 compares their explanation supervision and evaluation with XPlainVerse's scene-edit filtering and dual explanation styles [Sec. 3.1, Steps 3–5], human evaluation [Sec. 3.3], and perturbation protocol [Appendix F].

ResourceExplanation / reasoning emphasisEvaluation or construction emphasis
Veritas / HydraFakePattern-aware reasoning annotations, including planning and self-reflectionHierarchical generalization across architectures, forgery techniques and domains
Ivy-FakeExplanation annotations and reasoning chains for image/video AIGC detectionMultimodal training data and a manually verified evaluation set
XPlainVerseTechnical and simplified explanations for different readersInstruction-based scene edits, Edit-Check filtering, and clean/perturbed robustness evaluation

Table 6: Scope and emphasis of recent explainable detection benchmarks.

We will use these distinctions to expand Table 1 and Sec. 2, and explain how XPlainVerse's technical and simplified annotation styles differ from the structured reasoning annotations used by Veritas and Ivy-Fake.

References

[1] Hao Tan et al. Veritas: Generalizable Deepfake Detection via Pattern-Aware Reasoning. arXiv:2508.21048v2.

[2] Changjiang Jiang et al. Ivy-Fake: A Unified Explainable Framework and Benchmark for Image and Video AIGC Detection. arXiv:2506.00979v6.

Reviewer_b8uJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  1. Introduces a deepfake detection and explanation benchmark with one million images, including perturbed variants, and 1.53 million explanations.
  2. Contributions include edit filtering, technical and simple explanations, and entity/evidence agreement metrics.
  3. Validation includes human annotations, a user study, and detection experiments across seen and unseen generators.
Critical strengths
  1. Useful data resource. Large-scale explanations and edit-quality filtering support research beyond binary detection.
  2. Informative diagnostics. Cross-generator and region-swap experiments help explain failures on unseen generators.
Critical weaknesses
  1. Metric validity is unclear. Agreement between text judges does not establish agreement with human judgments of visual correctness. (Section 4; Appendix I.)
  2. Training is highly imbalanced. One generator supplies approximately 98.4% of training fakes, complicating interpretation of generalization failures. (Table 9.)
  3. References use privileged information. Their generation uses original images and edit instructions. The study does not clearly isolate whether all cited evidence is visible in the edited image alone.
  4. Robustness results are incomplete. Clean and perturbed test results are not separated, leaving the contribution of perturbations unclear.
Questions for authors
  1. How well do the proposed metrics correlate with human grounding judgments?
  2. How do the results change under generator-balanced training?
  3. Do reference explanations remain valid when raters see only the edited image?
  4. Can you report clean-versus-perturbed results and clarify whether perturbations receive new explanations?
3 comments
AuthorsOpenReview ↗

Response to Reviewer b8uJ (Part 1/2)

We sincerely thank the reviewer for their thoughtful comments and constructive suggestions. We appreciate the opportunity to further strengthen our evaluation through additional experiments and clarifications. We address each question below.

Q1. Agreement between automatic metrics and human judgments

Thank you for the question. To assess how well EntityScore and EvidenceScore align with human judgments of entity and evidence correspondence, we conducted an additional human study on 100 reference–prediction explanation pairs, where each pair was annotated by atleast two different users.

Participants saw explanations labelled A/B and counted, for each explanation, the total entities and evidence observations and how many were also present in the other explanation. Using the definitions in Appendix I, we calculated scores from each response and averaged the two human scores for each pair. Table 1 compares the automatic scores with these human-derived scores.

MetricPearson rrSpearman ρ\rhoMean absolute error
EntityScore0.8630.8330.128
EvidenceScore0.8700.8360.108

Table 1: Agreement with the mean human-derived score over 100 explanation pairs. Scores lie in [0,1][0,1].

Both metrics show high correlations and low absolute errors, indicating that their scores closely reflect human assessments of the shared entities and evidence on these pairs. This supports their intended role as reference-based explanation metrics [Sec. 4; Appendix I]. We will add these details in our paper as well.

Q2. Generator-balanced training

Sec. 5(Line 410-412) and Appendix J(Line 1930-1943, 2008-2015), report our generator-balanced training study with Qwen3.5-4B and BusterX++. The fake-image portion of this mixture covered five generators: Gemini-2.0-Flash, LongCat, Hunyuan, ICEdit, and GPT-Image-1. We excluded seven additional generators from fine-tuning and evaluated them as held-out generators: FLUX.1-dev, GPT-Image-1.5, Nano Banana 2, Nano Banana Pro, Qwen-Image-Edit, Seedream 4.5, and Wan 2.6.

We evaluated both models on a test subset containing images from all twelve generators. We calculated fake-image accuracy for each generator and then averaged the results separately for the five training generators and the seven held-out generators.

ModelOverall Acc.Fake F1Seen Fake Acc.Held-out Fake Acc.Gap
Qwen3.5-4B75.9%71.8%82.8%49.0%33.8 pp
BusterX++75.7%71.7%83.9%48.8%35.1 pp

For Qwen, mean fake-image accuracy on the same seven held-out generators is 49.0% in this experiment, compared with 15.3% in the original setting. Nevertheless, it remains 33.8 percentage points below accuracy on the five training generators. BusterX++ exhibits a similar gap of 35.1 points.

These results support the finding in Sec. 5 that strong performance on training generators does not ensure comparable performance on held-out generators. A substantial gap remains for both models even when the fine-tuning data span more generators.

AuthorsOpenReview ↗

Response to Reviewer b8uJ (Part 2/2)

Q3. Evaluation using only the edited image

Our human study in Sec. 3.3 and Appendix H presents participants with only the edited image and its explanation, without the original image, editing instruction or difference description. Across 118 images and 360 evaluations, the explanations receive 4.05/5 for grounding, 4.03 for specificity and 3.82 for completeness; unsupported claims receive 2.40/5 (lower is better), [Stage 3, Tables 17–18].

The anonymous A/B comparisons in Sec. 3.3, Table 4, and Appendix H (Stages 4–5) also use only the edited image. Our complex explanations receive 64.9% of grounding preferences versus 28.9% for SIDA-style explanations, with 6.2% ties. They are preferred for fewer unsupported claims in 54.9% of comparisons versus 31.1%, with 14.0% ties. Against human-written explanations, grounding preferences are 63.7% versus 23.0%, with 13.3% ties. Hence, these results support the validity of the reference explanations.

Q4. Clean-versus-perturbed results and explanation reuse

Perturbed images use the explanation of their clean counterpart. In the protocol described in Sec. 3.1 (Step 4) and Appendix F, perturbations simulate changes in image quality, such as compression and blur, while retaining the underlying edit. Keeping the reference explanation fixed allows us to assess robustness to these changes using the same annotation.

Table 2 provides an additional clean/perturbed analysis for BusterX++, Qwen3-VL-8B, InternVL3.5-14B, and Gemini-3-Flash. Using the ID/OOD definitions in Sec. 5, we report fake-class F1 separately for clean and perturbed images in both explanation settings.

SettingModelExplanationID CleanID PerturbedOOD CleanOOD Perturbed
Zero-shotBusterX++Complex59.7555.5048.2550.84
Zero-shotBusterX++Simple58.5054.6647.3250.45
Zero-shotGemini-3-FlashComplex76.0774.4572.1171.43
Zero-shotGemini-3-FlashSimple76.6373.5970.9069.25
Fine-tunedBusterX++Complex95.4088.4034.2839.77
Fine-tunedBusterX++Simple95.8489.0935.6539.92
Fine-tunedQwen3-VL-8BComplex97.7091.0631.6334.48
Fine-tunedQwen3-VL-8BSimple97.7291.7430.3735.02
Fine-tunedInternVL3.5-14BComplex85.8081.5541.1944.15
Fine-tunedInternVL3.5-14BSimple86.7682.5743.0646.51

Table 2: Fake-class F1 (%) on clean and perturbed images.

Across both explanation settings, all three fine-tuned models perform substantially better on clean ID images than on clean OOD images. The generator-sensitivity finding in Sec. 5 therefore persists when perturbed images are excluded. Perturbed ID images yield lower F1 for each model, while OOD F1 is slightly higher on the perturbed cohorts. Gemini retains its comparatively strong OOD performance on clean images.

We thank the reviewer again for their constructive feedback. In our rebuttal, we have included additional experiments on human agreement with our proposed metrics, generator-balanced training, and explanation grounding, along with clarifications addressing the remaining concerns.

We would greatly appreciate any further feedback on whether these results and clarifications address your concerns.

Reviewer_8iAbOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces XPlainVerse, a million-scale benchmark for explainable deepfake detection, comprising 470K real images, 530K manipulated images, and 1.53M image–explanation pairs. The authors propose an Edit-Check pipeline to filter low-quality manipulations and provide both technical and simplified explanations for different users. They further introduce EntityScore and EvidenceScore to evaluate the consistency of manipulated entities and supporting evidence between generated and reference explanations. Extensive experiments across multiple multimodal models demonstrate strong in-distribution performance but substantial degradation on unseen generators, highlighting the challenges of cross-generator generalization and evidence-grounded reasoning.

Critical strengths
  1. The paper addresses an important limitation of existing deepfake benchmarks by combining authenticity detection with natural-language explanation supervision at a substantially larger scale. The inclusion of both technical and simplified explanations is particularly meaningful for supporting users with different levels of expertise. Additionally, the Edit-Check pipeline demonstrates a systematic effort to improve data quality by filtering manipulations that fail to satisfy their intended editing instructions.

  2. The paper provides extensive evaluations across multiple multimodal models, including in-distribution and out-of-distribution detection, explanation quality, and robustness analysis. The substantial performance degradation on unseen generators is an important finding. Furthermore, the linear probing, frequency-spectrum analysis, and counterfactual region replacement experiments provide useful insights into models' reliance on generator-specific features rather than transferable forensic evidence.

Critical weaknesses
  1. Although the paper aims to construct visually grounded explanations, the explanation generation models have access to the original image, editing instructions, and visual-difference descriptions in addition to the manipulated image (Section 3.1). This may encourage models to generate plausible forensic descriptions based on editing metadata rather than independently observable artifacts. The Edit-Check pipeline primarily verifies whether manipulations are successfully performed, not whether every generated evidence claim is visually valid. While human evaluations are provided, stronger evidence-level validation is needed to establish the reliability of the generated explanations.

  2. The proposed metrics improve upon general text-similarity measures by evaluating entity and evidence-claim consistency. However, as described in Section 4 and Appendix I, they rely on matching predicted explanations against reference explanations without directly verifying the image content. Consequently, an incorrect visual claim appearing in both explanations may receive a high score, while valid evidence absent from the reference may be penalized. Since the reference explanations are also automatically generated, potential annotation biases may further affect metric reliability. Stronger validation against independently verified visual evidence is necessary to support the claimed grounding capability.

Questions for authors
  1. Since the explanation generation models receive the original image, editing instructions, and visual-difference descriptions, how do the authors ensure that the generated forensic claims are independently observable in the manipulated image? Could the authors quantify the proportion of visually supported versus hallucinated evidence claims using the human-annotated subset?

  2. How can EvidenceScore distinguish genuinely correct forensic explanations from explanations that match the reference text but describe nonexistent visual artifacts? Could the authors evaluate its reliability using deliberately constructed hallucinated explanations or independently verified evidence annotations?

  3. Appendix C shows that the training data is heavily dominated by Gemini-2.0-Flash, while several held-out generators also appear in the validation set. Were these validation samples used for model selection or hyperparameter tuning? Additionally, were all images derived from the same original source strictly separated across training, validation, and test splits?

  4. The human evaluation involves only 30 participants primarily recruited from institutional research groups (Appendix H). How representative are these participants of the intended non-expert users? Beyond subjective preferences and readability, is there evidence that simplified explanations improve users' understanding or identification of genuine manipulation evidence?

2 comments

We are sincerely grateful to the reviewer for taking the time to carefully assess our work. We greatly appreciate their thoughtful comments and constructive suggestions.

Q1. Visual grounding of reference explanations

The human evaluation in Sec. 3.3 and Appendix H (Stage 3, Tables 17–18) assesses whether our explanations are supported by the edited image alone. Participants see the edited image and its explanation, without the original image, edit instruction, or difference description. They rate grounding, specificity, completeness, convincingness, and unsupported claims.

The explanations receive 4.05/5 for grounding, 4.03 for specificity, 3.82 for completeness, and 3.80 for convincingness across 118 images and 360 evaluations. The unsupported-claims rating is 2.40/5 (where lower is better). The high grounding score therefore provides independent human evidence that participants can verify the explanations against visible image content, without needing the privileged context used during generation.

Q2. Scope of EntityScore and EvidenceScore

As described in Sec. 4 (Lines 349–366) and Appendix I (Lines 1666–1670), EntityScore and EvidenceScore evaluate whether predicted explanations correctly identify the entities and supporting evidence described in the reference explanations, rather than relying solely on semantic similarity (e.g., BERTScore). These metrics are designed to assess forensic-content agreement with the references, not to independently verify whether the described artifacts are visually present in the image.

For example, a reference may state, “The person on the left has an extra finger.” A prediction stating, “The person on the right has an extra finger,” changes the entity, while “The person on the left has a missing finger” changes the evidence. These sentences use almost identical wording, so a general semantic similarity metric such as BERTScore may still assign them a high score despite the mismatch in forensic content. EntityScore checks whether the prediction identifies the same diagnostic entity (in this example, the same person's hand), while EvidenceScore checks whether it recovers the specific claim about an extra finger [Appendix I, Lines 1745–1751].

Q3. Data splits, model selection, and generator balance

Each clean image and its perturbed counterpart are kept in the same split. For the protocol in Sec. 5 and Appendix C, we chose to define ID/OOD by generator, without additionally requiring all edits of a source image to remain in one split, as different generators can introduce different visual artifacts, and our aim is to test whether detectors generalize beyond the generators represented during fine-tuning. Different edits of the same image can therefore occur across splits.

Yes, validation samples were used for model selection, with the checkpoint selected based on the lowest validation loss. The validation set includes all six generators, including Hunyuan, LongCat, and Qwen-Image-Edit, which are excluded from fine-tuning updates but included in checkpoint selection [Appendix C, Table 9]. We will add checkpoint-selection details in Appendix J.

Regarding Gemini's dominance, the study in Sec. 5 and Appendix J (Table 24) uses a more balanced training mixture, with fake images from five generators: Gemini-2.0-Flash, LongCat, Hunyuan, ICEdit, and GPT-Image-1. This reduces Gemini's share of fake images from 98.4% to 45.1% and balances the real and fake classes, while substantially reducing the generator imbalance. Qwen's mean recall on the same seven held-out generators rises from 15.3% to 49.0%, but remains 33.8 percentage points below the mean for training generators. BusterX++ retains a similar gap of 35.1 points. Balanced training therefore improves transfer, but a substantial gap between training-exposed and held-out generators remains.

Q4. Participant diversity and accessibility of simple explanations

Although the 30 participants were recruited through institutional networks, they varied across academic disciplines, countries, and cultural backgrounds. The group included participants from both technical and nontechnical courses, so the evaluation in Sec. 3.3 and Appendix H includes perspectives from users without a technical background. We will expand the participant description in Appendix H to make this composition clear, while retaining the limitations of institutional recruitment discussed in Appendix B.

Regarding understanding, across 360 evaluations of 119 images, simple explanations receive 3.99/5 for immediate understanding, 3.97 for memorability, and 2.13 for effort (lower effort is better).

For respondents who report a nontechnical background, immediate understanding is 4.43/5 and effort is 1.66/5 across 114 evaluations of 59 images. These responses support the accessibility of simple explanations for participants without technical training. We will include this breakdown of the existing study in Appendix H.

We thank the reviewer again for their thoughtful feedback. We have provided detailed responses to the concerns raised, including additional analyses and clarifications.

As the discussion period approaches its conclusion, we would greatly appreciate hearing whether our responses have addressed your main concerns, or if any questions remain.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.