- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces FraudBench, a multimodal benchmark for detecting AI-generated fraudulent refund evidence in e-commerce and related transaction scenarios. The authors collect real-world review data from four sources and use 12 image generation and editing models to synthesize fraudulent damage evidence, along with corresponding fake review texts. The benchmark further defines five evaluation dimensions to assess the reliability of detection methods under different settings. The authors evaluate MLLMs and specialized AI-generated image detectors. Experimental results show that MLLMs generally preserve genuine evidence but fail to detect most synthetic damage, while specialized detectors achieve better fake-image detection performance but exhibit substantial sensitivity to generators and model checkpoints, with some producing high FP rates on genuine evidence.
Critical strengths
- FraudBench addresses a practical and important refund fraud problem in real-world e-commerce scenarios.
- The benchmark goes beyond single-image detection by introducing a claim-centered perspective on evidence verification. Images submitted in refund scenarios should be evaluated in relation to the specific claims they support, rather than being treated merely as independent Real/Fake images.
- The benchmark considers several important aspects of realistic refund scenarios, including diverse review contexts (e.g., travel services, food delivery, dine-in, and pickup), multilingual reviews, and both single-image and multi-image.
- The synthetic evidence generation pipeline is well-motivated. Rather than generating fake-damaged samples arbitrarily, the authors first mine damage patterns from real damaged evidence and then synthesize defects based on these patterns. This improves the reliability of FraudBench.
Critical weaknesses
-
The evaluation metrics mainly focus on the final verdict, with limited attention to the quality and faithfulness of the explanations generated by MLLMs. In practical refund verification, whether the provided explanation accurately supports the decision is also important.
-
The authors argue that refund fraud detection differs from conventional AI-generated image detection by additionally requiring multi-view visual consistency and alignment between visual evidence and the user's stated claim. However, the fake review texts are generated based on the corresponding synthetic damage evidence, making the images and claims generally aligned by construction. Consequently, the benchmark does not sufficiently evaluate cases where visual evidence contradicts the user's claim. Although Appendix D.4.3 includes an image-text mismatch experiment, the mismatched reviews are sampled from different product categories, rather than more challenging cases involving the same product but inconsistent damage claims. This somewhat limits the distinction between the proposed task and conventional AI-generated image detection.
-
Specialized detectors such as CO-SPY [ProGAN] and ForgeLens [GenImage] achieve relatively strong results without using textual information, while MLLMs show only modest improvements when review text is provided. This raises the question of whether the main challenge in fraudulent refund evidence detection still lies in visual authenticity detection, and to what extent claim-centered information provides additional benefits. A more detailed analysis of the contribution of claim information would help better establish the value of the proposed claim-centered evaluation framework.
Questions for authors
As discussed in the second and third points of Critical Weaknesses, I have concerns about the necessity of a claim-centered perspective in fraudulent refund evidence detection, as the authenticity of the evidence images appears to be more important.
5 comments
Response to Reviewer ASFd (2/2)
Multiple Images:
| Complaint Shown with Images | GPT-6 Astra | GPT-5.6 Terra | Gemini 3.8 Flash | Grok 4.6 | DeepSeek V4.1 Flash | Kimi K3 | Qwen3.8-Max | Average |
|---|---|---|---|---|---|---|---|---|
| None | 0.773 | 0.649 | 0.612 | 0.570 | 0.637 | 0.563 | 0.694 | 0.642 |
| Genuine, Other Category | 0.770 | 0.653 | 0.610 | 0.570 | 0.715 | 0.571 | 0.667 | 0.651 |
| Genuine, Same Category, Matched Damage | 0.781 | 0.635 | 0.615 | 0.567 | 0.705 | 0.564 | 0.641 | 0.644 |
| Genuine, Same Category, Contradicting Damage | 0.785 | 0.648 | 0.611 | 0.566 | 0.718 | 0.570 | 0.655 | 0.651 |
Contradicting claims barely change the result (0.600 vs. 0.603 with single images, and 0.651 vs. 0.644 with multiple images). With single images, they make three models somewhat more suspicious of both classes (TNR from 0.992 to 0.970 and TPR from 0.214 to 0.229 on average), which does not help separate them. When asked whether an image is authentic, current MLLMs do not check it against the claim.
2. Claim Verification with Same-Product Contradicting Claims
When asked directly, MLLMs check the claim well. Each of 208 real-damaged reviews is shown twice with the same photos: once with its complaint reworded by an LLM, and once with the complaint rewritten by the same LLM to describe a clearly different type of damage to the same product (for example, a cracked drawer claimed to be burnt). Reviews whose complaint names several damage types were excluded. The models are asked whether the photos show the problem described.
Since all photos are authentic and both complaints are LLM-written, neither an image-only detector nor a writing-style cue can do better than chance (0.5). Across 14 MLLMs, balanced accuracy is 0.871 on average (0.789 to 0.944) with an AUROC of 0.921: the models accept 91% of the matching complaints and reject 84% of the contradicting ones. We will add this control to the benchmark as a claim-verification track.
Weakness 3: Is Visual Authenticity the Main Challenge?
"...This raises the question of whether the main challenge in fraudulent refund evidence detection still lies in visual authenticity detection, and to what extent claim-centered information provides additional benefits. ..."
Response 3:
Yes, visual authenticity remains the main challenge, and the claim adds a different kind of benefit.
For deciding whether the damage is AI-generated, adding a genuine claim changes accuracy by less than 0.01. For deciding whether the photos support the claim, the claim gives image-only detectors nothing, while it raises MLLMs from chance to 0.871. Balanced accuracy with and without the claim, on the same images (from the tables in Response 2):
| Question | Models | Image Only | Image + Claim | Gain from the Claim |
|---|---|---|---|---|
| Is the Damage AI-Generated? (Single Image) | 12 MLLMs | 0.608 | 0.603 | −0.005 |
| Is the Damage AI-Generated? (Multiple Images) | 7 MLLMs | 0.642 | 0.644 | +0.002 |
| Do the Photos Show the Claimed Problem? | Specialized Detectors | 0.500 | Cannot Read the Claim | -- |
| Do the Photos Show the Claimed Problem? | 14 MLLMs | 0.500 | 0.871 | +0.371 |
1. Visual Authenticity Is the Main Challenge
In the first two rows, each image receives a genuine 1-star complaint from the same category that describes the same damage type, so the text carries no label information. Accuracy does not change, so deciding whether the damage is fabricated depends on the images. This image task remains hard: in the paper's single-image results (Table 1), the 16 MLLMs reach a balanced accuracy of only 0.584 on average and detect only 23.0% of the fake-damaged images.
2. The Claim Adds a Check That Only MLLMs Can Perform
In the claim-verification control, the same authentic photos are shown once with a matching and once with a contradicting complaint, so without the claim the two cases cannot be told apart (0.500 by construction). With the claim, 14 MLLMs reach 0.871 on average (AUROC 0.921) over 208 reviews. Specialized detectors take only the image, so the claim gives them no benefit.
We will state this scope clearly in the revision and add the claim-verification control to the benchmark as its own track.
Thank you again for your valuable comments! We hope that our clarifications and new results address your concerns, and we would be glad to continue the discussion if any questions remain.
Authors of Submission #6087
I appreciate the authors' clarification and the additional experiments. However, the submitted PDF appears unchanged: it does not include these new results or the corresponding revisions. As these additions are essential to the paper's completeness. This may affect my rating.
Response to Reviewer ASFd (1/2)
We sincerely appreciate your recognition of our contributions, in particular the practical importance of the refund-fraud problem and the claim-centered perspective. Your questions led to some of the most informative new experiments in this rebuttal. We would like to address each concern below.
Weakness 1: Quality and Faithfulness of Explanations
Response 1:
Thank you for this suggestion. We have added two analyses of explanation quality, and both show that most correct verdicts point at the actual edit. Together with the localization study in the appendix, explanations are now evaluated in three ways:
1. Explanations Name the Right Damage (New)
For fake-damaged images that a model correctly flags (single image, no review text), we checked whether its explanation names the damage type that was actually synthesized. Across seven models, this holds for 68% to 83% of correct verdicts (76% on average). When the damage labels are shuffled among the same model's correct verdicts within each category, the rate falls to 37% to 51% (44% on average).
2. Explanations Point to the Edited Region (New)
We sampled 300 of these correct verdicts (50 per generator) and outlined the edited region on each image. Two MLLM judges that are not among the seven models (GPT-6 Sol and Qwen3.8-Flash) then decided whether the explanation refers to something inside the outline. As controls, the same explanation was checked against the same outline moved to another part of the image, and the true outline was checked against the explanation of a different image:
The two judges agree on 96% of the verdicts (Cohen's κ = 0.79). The rate is high for every generator (0.77 to 0.97, against 0.16 to 0.46 for the moved outline), and lowest for GPT Image 2, which is also the hardest generator to detect. The judges also find the cited evidence visible in the image for 98% of the explanations, against 29% when the explanation of another image is used. Most correct verdicts therefore point at the edited defect, not only at generic artifacts.
3. Localization (Appendix D.4.7)
Boxes predicted by Gemini 3 Flash reach a mean IoU of 0.208 with the edited region, compared with 0.087 for FakeShield masks on the same 786 images, although part of this advantage comes from the boxes being coarser.
We would add these explanation metrics to the evaluation section.
Weakness 2: Image–Claim Alignment by Construction
Response 2:
We ran the same-category mismatch you suggested. MLLMs ignore the claim when asked whether an image is authentic, but they check it well when asked directly whether the photos support it.
1. Contradicting Claims Under the Authenticity Question
Every review of both classes receives a genuine 1-star complaint written for another product in the same category. The complaint describes either the same damage type as the image (matched) or a different one (contradicting). Since both classes draw from the same pool of genuine text, the text carries no label information, and the two conditions differ only in whether the claim agrees with the image. The single-image results cover 410 images (364 fake and 46 real) from the paper's stratified sample with 12 MLLMs, and the multi-image results cover all 1,668 reviews with at least two images with the seven main MLLMs. Both tables report balanced accuracy; the "None" rows are the runs reported in the paper, and the other rows are new runs from the same week.
Single Image: