- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This work presents Mandela-Bench, a novel benchmark for knowledge-driven image forgery detection, targeting a critical blind spot of existing multimodal evaluation protocols. Different from traditional forgery detection benchmarks that rely on pixel artifacts, visual anomalies or explicit text-image mismatches, Mandela-Bench focuses on visually plausible fake images where forgery judgment strictly depends on real-world factual knowledge. The benchmark consists of 1,981 images (1,507 edited and 474 authentic) covering six types of factual contradictions (person, object, place, visible text, event attribute, artwork) plus a non-contradictory K0 control group. With systematic controlled experiments across 36 multimodal large language models (MLLMs), the paper reveals a key failure pattern: MLLMs often rely on memorized canonical scene knowledge rather than verifying local visual evidence. Even when models possess correct world knowledge and can perceive local facial/object discrepancies, they frequently misjudge factually tampered classic images as authentic due to memory-based reconstruction. The authors also propose a fine-grained evaluation metric, Knowledge-Grounded Rate (KGR), to distinguish correct forgery judgments driven by valid factual reasoning from random guesses or superficial visual cues.
Critical strengths
-
Novel and valuable research perspective. This work addresses an understudied core problem of MLLM visual verification: the inconsistency among model’s world knowledge, local visual perception, and final factual validation. It shifts forgery detection evaluation from low-level artifact detection to high-level knowledge-driven reasoning, filling a crucial gap in existing multimodal robustness benchmarks.
-
Rigorous dataset and evaluation design. The well-designed K0 control group effectively eliminates the interference of general editing artifacts, enabling precise isolation of factual contradiction detection capability. The proposed E0-E4 evidence hierarchy and KGR metric innovatively evaluate the rationality and grounding of model predictions, rather than only final classification accuracy.
-
Comprehensive and credible experimental validation. The paper adopts multi-angle probing experiments (TEXT, FACE, Caption/Crop intervention, absent-celebrity test, etc.) to rule out alternative explanations. Evaluations on 36 models of diverse scales and types verify the universality of the observed memory reconstruction failure mode.
-
Reproducible and transparent research. The work provides detailed dataset construction rules, prompt settings, automatic scoring calibration, and human evaluation results, laying a solid foundation for follow-up research.
Critical weaknesses
-
Limited benchmark generalization. The dataset only covers highly recognizable classic historical images, celebrities and fixed single-fact tampering scenarios, which cannot represent real-world complex forgeries involving ordinary people, unfamiliar scenes, multi-fact edits or controversial events.
-
Incomplete elimination of visual shortcuts. Although mainstream pixel-level forgery detectors fail on this benchmark, it cannot fully exclude potential subtle visual or editing-specific artifacts that may affect model judgments, making the "pure knowledge-driven detection" claim not absolute.
-
Confounding variables in core intervention experiments. The key crop-based de-recognition experiments alter multiple variables (global scene context, target saliency, image composition) simultaneously besides reducing canonical scene recognition. Thus, the causal link between memory reconstruction and verification failure cannot be strictly proven.
-
Ambiguity in metric and mechanism interpretation. The KGR metric allows credit for abnormal entity recognition without complete factual reasoning, which overestimates models’ rigorous verification ability. Moreover, the internal "memory reconstruction" mechanism is only inferred from external model behaviors, lacking direct causal or internal representation evidence.
-
Lack of effective solutions. This work only systematically diagnoses MLLMs’ factual verification failures but does not propose targeted optimization strategies to mitigate the memory override issue, limiting its practical improvement value.
-
Statistical uncertainty. Key conclusions rely on small subsets of samples, without systematic confidence interval and effect size analysis, leading to limited stability of partial findings.
Questions for authors
Please refer to the weaknesses.
5 comments
response part 2
3. Confounding Variables in De-Recognition and Converging Non-Crop Evidence (Response to W3)
Recognizing the scene still lowers detection when image composition and target saliency are controlled. Pad+Caption keeps the image size, the edited region's size and position, and the event caption identical to Full+Caption and masks only the surrounding scene: scene recognition falls by 53–73 points, and KGR and E4 rise on all three tested models. Without any image modification, recognizing the scene in Full still lowers detection after controlling for edit saliency (Appendix I, Table 10). Item 4 states what the two analyses leave open.
- Scale-matched control: On 145 peripheral forgeries and 98 authentic counterparts, Pad+Caption keeps every pixel inside the Crop box at native resolution and original coordinates (243/243 pixel-verified) and uses the same event caption as Full+Caption; only the surrounding scene becomes neutral grey (RGB 128). KGR / E4 rise from 13.10 / 9.66 to 16.55 / 13.10 on GPT-5.6 (Det ), from 2.07 / 1.38 to 11.72 / 8.97 on Kimi-K3 (KGR , E4 ), and from 1.38 / 0.69 to 6.21 / 3.45 on Qwen3.6-27B (KGR , Det ).
- Converging evidence without image modification: A scene-clustered logistic regression over 13,660 model–item observations in the unmodified Full condition, controlling for edit saliency (C1 main figure, C2 secondary figure, C3 background), knowledge family, and domain, shows that scene recognition lowers the odds of detecting a people-family forgery (pooled OR = 0.65, 95% CI [0.51, 0.82], ; negative for 20 of 24 recognizing models; Appendix I, Table 10).
- Same caption: Crop+Caption and Full+Caption supply the identical event caption, so the Crop+Caption gain (KGR 1.3×–8.9× that of Full+Caption across all 21 recognizing models) does not come from added text.
- Scope: Pad+Caption rules out magnification and coordinate shifts, and the within-Full regression leaves the image unchanged while controlling for coarse saliency. Replacing a cluttered scene with uniform grey can still raise local target salience, and C1/C2/C3 labels do not capture all within-image salience variation. We therefore present these results as converging behavioral evidence that scene recognition biases verification and note the salience confound in the Limitations section.
Full comparison in %. All conditions except Crop+Caption use the 1280 px canvas at native target scale; Crop+Caption uses the native-resolution crop (median 320 px). : exact McNemar vs. Full / Full+Caption (FC).
| Model | Condition | RECOG (fake / real) | Det | Auth | Bal | KGR | E4 | vs. Full / FC |
|---|---|---|---|---|---|---|---|---|
| GPT-5.6 | Full (L1) | 83.45 / 86.73 | 53.79 | 84.69 | 69.24 | 14.48 | 7.59 | — |
| GPT-5.6 | Full+Caption | 83.45 / 86.73 | 57.24 | 85.71 | 71.48 | 13.10 | 9.66 | — |
| GPT-5.6 | Crop+Caption | — | 69.66 | 86.73 | 78.19 | 16.55 | 13.79 | Det 0.0022 / 0.0096 |
| GPT-5.6 | Pad | 30.34 / 38.78 | 64.83 | 69.39 | 67.11 | 15.17 | 6.90 | Det 0.0764 / 0.2000 |
| GPT-5.6 | Pad+Caption | 30.34 / 38.78 | 69.66 | 78.57 | 74.11 | 16.55 | 13.10 | Det 0.0011 / 0.0064; E4 vs. Full 0.0386 |
| Kimi-K3 | Full (L1) | 88.28 / 87.76 | 35.86 | 92.86 | 64.36 | 3.45 | 2.07 | — |
| Kimi-K3 | Full+Caption | 88.28 / 87.76 | 26.90 | 95.92 | 61.41 | 2.07 | 1.38 | — |
| Kimi-K3 | Crop+Caption | — | 62.76 | 86.73 | 74.75 | 8.28 | 6.90 | Det / |
| Kimi-K3 | Pad | 15.17 / 15.31 | 15.17 | 94.90 | 55.04 | 3.45 | 1.38 | Det / 0.0060 |
| Kimi-K3 | Pad+Caption | 15.17 / 15.31 | 33.10 | 94.90 | 64.00 | 11.72 | 8.97 | KGR 0.0005 / 0.0001; E4 0.0020 / 0.0010 |
| Qwen3.6-27B | Full (L1) | 73.79 / 77.55 | 40.69 | 56.12 | 48.41 | 2.76 | 1.38 | — |
| Qwen3.6-27B | Full+Caption | 73.79 / 77.55 | 31.72 | 65.31 | 48.52 | 1.38 | 0.69 | — |
| Qwen3.6-27B | Crop+Caption | — | 51.72 | 68.37 | 60.05 | 5.52 | 3.45 | Det 0.0479 / |
| Qwen3.6-27B | Pad | 16.55 / 15.31 | 89.66 | 12.24 | 50.95 | 8.28 | 0.69 | Det / |
| Qwen3.6-27B | Pad+Caption | 16.55 / 15.31 | 56.55 | 66.33 | 61.44 | 6.21 | 3.45 | Det 0.0018 / ; All 0.0012; KGR vs. FC 0.0156 |
response part 3
4. KGR vs. E4 Metric Strictness and Mechanism Interpretation (Response to W4)
We report KGR (E3+) and E4 separately: KGR credits naming the altered entity or fact, and E4 additionally requires explaining why it contradicts the recorded event. Appendix K (Table 15) reports both for all 36 models, and their difference on the same denominator quantifies the credit KGR assigns beyond strict verification.
- KGR credit without contradiction reasoning: On the same 1,507 forgeries under L1, Gemini-3.8-Flash reaches 55.14% KGR and 37.16% E4. The 17.98-point gap consists of responses that identify the altered entity without stating the specific contradiction, the case your comment describes. Main-text Table 2 will report E4 alongside KGR to separate entity grounding from complete contradiction reasoning.
- Behavioral scope of "reconstruction from memory": Our evidence is behavioral. On absent-celebrity pairs where a model passes the isolated FACE probe (rejecting the replacement face on a neutral canvas), it still judges the full photograph authentic 63.6% (Kimi-K3), 69.8% (Qwen3.6-27B), and 29.6% (Gemini-3.8-Flash) of the time, and 34 of 126 such REAL responses (27.0%) name the erased person as present. Naming an erased person while judging the photograph authentic is behavioral evidence that familiar-scene priors interfere with verification. Locating the responsible internal circuit would require white-box activation analysis; the strongest models on the benchmark (Gemini-3.8-Flash, GPT-5.6) are proprietary, and mechanistic probing of open-weight models is a separate study. §4.1 accordingly introduces naming the erased person as a behavioral signature of reconstruction from memory.
5. Practical Mitigation Strategies (Response to W5)
The diagnosis suggests keeping scene recognition but separating it from local verification. A training-free two-stage workflow built on this idea improves detection and E4 on all three tested models without a ground-truth caption.
- Mechanism: Scene recognition supplies the event context needed for verification but also suppresses local checking. Adding the true caption to the full image gives no KGR gain, whereas pairing the same caption with the local crop raises KGR to 1.3×–8.9× across all 21 recognizing models (Table 3).
- Workflow: Stage 1 queries the full image and keeps the model's own scene description; Stage 2 supplies the local crop with that description and asks the model to verify it. Neither stage uses a ground-truth caption. We use the benchmark's annotated region as the crop; a deployed system would take candidate boxes from an off-the-shelf person or object detector.
- Effect: On 145 peripheral people-family forgeries and 98 authentic originals, detection rises over Full L1 by +13.80 to +26.90 points on all three models (), and E4 rises on all three (7.59 → 8.97 on GPT-5.6, 2.07 → 7.59 on Kimi-K3 [], 1.38 → 3.45 on Qwen3.6-27B). KGR rises on Kimi-K3 (3.45 → 12.41) and Qwen3.6-27B (2.76 → 8.28) and is unchanged on GPT-5.6, whose higher detection comes with lower authentic accuracy (84.69 → 69.39), for a 1.7-point gain in balanced accuracy.
| Model | Condition | Det | Auth | Bal | KGR | E4 | Exact McNemar vs. Full L1 |
|---|---|---|---|---|---|---|---|
| GPT-5.6 | Full (L0) | 28.28 | 95.92 | 62.10 | 4.14 | 1.38 | — |
| GPT-5.6 | Full (L1) | 53.79 | 84.69 | 69.24 | 14.48 | 7.59 | — |
| GPT-5.6 | Self-Captioned Crop | 72.41 | 69.39 | 70.90 | 14.48 | 8.97 | Det 0.00014 |
| Kimi-K3 | Full (L0) | 17.93 | 95.92 | 56.92 | 1.38 | 0.69 | — |
| Kimi-K3 | Full (L1) | 35.86 | 92.86 | 64.36 | 3.45 | 2.07 | — |
| Kimi-K3 | Self-Captioned Crop | 49.66 | 89.80 | 69.73 | 12.41 | 7.59 | Det 0.0066; KGR 0.00024; E4 0.0215 |
| Qwen3.6-27B | Full (L0) | 29.66 | 77.55 | 53.60 | 2.07 | 0.69 | — |
| Qwen3.6-27B | Full (L1) | 40.69 | 56.12 | 48.41 | 2.76 | 1.38 | — |
| Qwen3.6-27B | Self-Captioned Crop | 67.59 | 55.10 | 61.34 | 8.28 | 3.45 | Det ; KGR 0.0215 |
- Baseline for future work: E4 stays below 10% on all three models, so the workflow is a training-free baseline rather than a complete solution. It supports decoupling global scene recognition from local verification at inference time, and Mandela-Bench's matched K0 controls and E0–E4 grading provide a standardized testbed for measuring further improvements.
response part 1
Thank you for your review and support of Mandela-Bench's research perspective and evaluation design. We address your six comments below using the paired K0 controls, the E0–E4 evidence hierarchy, the multi-probe evaluation across 36 MLLMs, and new rebuttal experiments.
1. Benchmark Scope and Generalization (Response to W1)
We agree that the benchmark does not cover forgeries of ordinary people, unfamiliar scenes, or multi-fact edits, and this scope is deliberate. Knowledge-only verification needs a fact the model can check from memory, which canonical scenes and single verifiable contradictions provide in our single-image, retrieval-free setting (§3.1, Appendices A–B); the other cases require external references.
2. Controlling for Subtle Visual or Editor-Specific Shortcuts (Response to W2)
Shared pipeline artifacts cannot explain the detection gap in our paired controls: K0 edits and celebrity swaps go through the same editing and re-encoding pipeline, yet models rarely flag K0 and frequently flag swaps (§3.1, Appendix C, Appendix L Table 20).
On the same seed images, K0 items and knowledge-anchored forgeries use the same editor (GPT Image 2), operation types, and JPEG Q92 1280 px re-encoding; only K0 lacks a factual contradiction. If shared traces drove detection, K0 would be flagged at comparable rates. It is not:
Swaps and K0 edits insert different individuals, so their local boundary cues can still differ; the regressions control for the edit-specific cues that pixel forensic detectors capture. The manuscript will state that these controls isolate factual verification relative to matched K0 edits and forensic detectors, rather than establishing that no item carries a local cue.