acceptodds
Under review as a conference paper at ICLR 2027

Mandela-Bench: Multimodal Models Remember Canonical Images Instead of Seeing Them

Abstract

Historical photographs and other canonical images can now be edited seamlessly with a single instruction, often leaving no reliable pixel-level trace. In such cases, the only evidence of manipulation may be a fact about what the image depicts. Existing benchmarks instead rely on generator artefacts, image–caption inconsistencies, visual implausibilities, or external references, and therefore do not test whether a model can use its own world knowledge to verify a recognized image. We introduce Mandela-Bench, containing 1,507 edits of canonical images: 1,359 knowledge-only forgeries, each contradicting one verifiable fact, and 148 anchor-free controls that preserve the editing process without introducing a factual contradiction, together with 474 untouched originals. We score not only whether a model detects a forgery, but whether its explanation identifies the inserted entity or the fact being violated. Across 36 multimodal models, from 0.8B parameters to frontier scale, we find a consistent failure mode. When a public figure is removed from a familiar photograph, models still name that person in up to 72.7% of responses. Some models can distinguish the replacement face from the original when shown in isolation, yet still judge the full edited photograph as authentic. Providing the true event and date does not improve knowledge-grounded detection, whereas providing the same information after cropping away the recognizable composition does. Even under explicit verification prompts, only one of the 36 models meets the KGR criterion on at least half of the forged images. These results suggest that the failures cannot be explained by missing knowledge or inadequate perception alone. Instead, they are consistent with recognition biasing verification toward the remembered canonical image rather than the observed edit.

Primary area:
datasets and benchmarks
Keywords:
image forgery detection, knowledge-grounded evaluation, multimodal large language models, vision-language models, benchmark

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 5 reviews, above 85% of the others.
Reviewer_4yrkOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This work presents Mandela-Bench, a novel benchmark for knowledge-driven image forgery detection, targeting a critical blind spot of existing multimodal evaluation protocols. Different from traditional forgery detection benchmarks that rely on pixel artifacts, visual anomalies or explicit text-image mismatches, Mandela-Bench focuses on visually plausible fake images where forgery judgment strictly depends on real-world factual knowledge. The benchmark consists of 1,981 images (1,507 edited and 474 authentic) covering six types of factual contradictions (person, object, place, visible text, event attribute, artwork) plus a non-contradictory K0 control group. With systematic controlled experiments across 36 multimodal large language models (MLLMs), the paper reveals a key failure pattern: MLLMs often rely on memorized canonical scene knowledge rather than verifying local visual evidence. Even when models possess correct world knowledge and can perceive local facial/object discrepancies, they frequently misjudge factually tampered classic images as authentic due to memory-based reconstruction. The authors also propose a fine-grained evaluation metric, Knowledge-Grounded Rate (KGR), to distinguish correct forgery judgments driven by valid factual reasoning from random guesses or superficial visual cues.

Critical strengths
  • Novel and valuable research perspective. This work addresses an understudied core problem of MLLM visual verification: the inconsistency among model’s world knowledge, local visual perception, and final factual validation. It shifts forgery detection evaluation from low-level artifact detection to high-level knowledge-driven reasoning, filling a crucial gap in existing multimodal robustness benchmarks.

  • Rigorous dataset and evaluation design. The well-designed K0 control group effectively eliminates the interference of general editing artifacts, enabling precise isolation of factual contradiction detection capability. The proposed E0-E4 evidence hierarchy and KGR metric innovatively evaluate the rationality and grounding of model predictions, rather than only final classification accuracy.

  • Comprehensive and credible experimental validation. The paper adopts multi-angle probing experiments (TEXT, FACE, Caption/Crop intervention, absent-celebrity test, etc.) to rule out alternative explanations. Evaluations on 36 models of diverse scales and types verify the universality of the observed memory reconstruction failure mode.

  • Reproducible and transparent research. The work provides detailed dataset construction rules, prompt settings, automatic scoring calibration, and human evaluation results, laying a solid foundation for follow-up research.

Critical weaknesses
  • Limited benchmark generalization. The dataset only covers highly recognizable classic historical images, celebrities and fixed single-fact tampering scenarios, which cannot represent real-world complex forgeries involving ordinary people, unfamiliar scenes, multi-fact edits or controversial events.

  • Incomplete elimination of visual shortcuts. Although mainstream pixel-level forgery detectors fail on this benchmark, it cannot fully exclude potential subtle visual or editing-specific artifacts that may affect model judgments, making the "pure knowledge-driven detection" claim not absolute.

  • Confounding variables in core intervention experiments. The key crop-based de-recognition experiments alter multiple variables (global scene context, target saliency, image composition) simultaneously besides reducing canonical scene recognition. Thus, the causal link between memory reconstruction and verification failure cannot be strictly proven.

  • Ambiguity in metric and mechanism interpretation. The KGR metric allows credit for abnormal entity recognition without complete factual reasoning, which overestimates models’ rigorous verification ability. Moreover, the internal "memory reconstruction" mechanism is only inferred from external model behaviors, lacking direct causal or internal representation evidence.

  • Lack of effective solutions. This work only systematically diagnoses MLLMs’ factual verification failures but does not propose targeted optimization strategies to mitigate the memory override issue, limiting its practical improvement value.

  • Statistical uncertainty. Key conclusions rely on small subsets of samples, without systematic confidence interval and effect size analysis, leading to limited stability of partial findings.

Questions for authors

Please refer to the weaknesses.

5 comments
AuthorsOpenReview ↗

response part 1

Thank you for your review and support of Mandela-Bench's research perspective and evaluation design. We address your six comments below using the paired K0 controls, the E0–E4 evidence hierarchy, the multi-probe evaluation across 36 MLLMs, and new rebuttal experiments.

1. Benchmark Scope and Generalization (Response to W1)

We agree that the benchmark does not cover forgeries of ordinary people, unfamiliar scenes, or multi-fact edits, and this scope is deliberate. Knowledge-only verification needs a fact the model can check from memory, which canonical scenes and single verifiable contradictions provide in our single-image, retrieval-free setting (§3.1, Appendices A–B); the other cases require external references.

  • Why canonical images and single-fact edits: For an unfamiliar scene or a private individual, the model holds no fact about who or what should be present. Retrieval-assisted settings such as LookupForensics and REVEAL address a different problem (Appendix A). Anchoring each item to one Wikidata-verifiable contradiction (Appendix B) keeps the ground truth objective and non-controversial and attributes each failure to a specific knowledge family rather than to multi-fact interactions.
  • Diverse domains: Within this scope, the 1,981 items span six knowledge families (people, objects, places, visible text, event attributes, artwork) and six visual domains (historical photographs, film stills, sports, paintings, political news, meme templates; Appendix Figure 5, Table 13).

2. Controlling for Subtle Visual or Editor-Specific Shortcuts (Response to W2)

Shared pipeline artifacts cannot explain the detection gap in our paired controls: K0 edits and celebrity swaps go through the same editing and re-encoding pipeline, yet models rarely flag K0 and frequently flag swaps (§3.1, Appendix C, Appendix L Table 20).

On the same seed images, K0 items and knowledge-anchored forgeries use the same editor (GPT Image 2), operation types, and JPEG Q92 1280 px re-encoding; only K0 lacks a factual contradiction. If shared traces drove detection, K0 would be flagged at comparable rates. It is not:

  1. Across all 36 models, K0 detection is at most 7 points above the false-alarm rate on untouched authentic images, and below it for 26 of 36 models (§5, Table 2). In our 60-scene two-stage human evaluation, humans flag 0.0% (0/12) of K0 items in both stages.
  2. On identical source images (same-seed triads, Table 3), K0 detection is 2.4% for Gemini-3.8-Flash, GPT-5.6, and Kimi-K3, versus 71.8%, 52.6%, and 70.5% for celebrity swaps, a 50–69 point gap.
  3. Seed-clustered logistic regressions controlling for log edit footprint and standardized NPR (AUC 0.577) and UniversalFakeDetect (AUC 0.538) scores keep the knowledge contrast significant for 16 of 17 models (p<0.05p < 0.05; Appendix L, Table 20).

Swaps and K0 edits insert different individuals, so their local boundary cues can still differ; the regressions control for the edit-specific cues that pixel forensic detectors capture. The manuscript will state that these controls isolate factual verification relative to matched K0 edits and forensic detectors, rather than establishing that no item carries a local cue.

AuthorsOpenReview ↗

response part 2

3. Confounding Variables in De-Recognition and Converging Non-Crop Evidence (Response to W3)

Recognizing the scene still lowers detection when image composition and target saliency are controlled. Pad+Caption keeps the image size, the edited region's size and position, and the event caption identical to Full+Caption and masks only the surrounding scene: scene recognition falls by 53–73 points, and KGR and E4 rise on all three tested models. Without any image modification, recognizing the scene in Full still lowers detection after controlling for edit saliency (Appendix I, Table 10). Item 4 states what the two analyses leave open.

  1. Scale-matched control: On 145 peripheral forgeries and 98 authentic counterparts, Pad+Caption keeps every pixel inside the Crop box at native resolution and original coordinates (243/243 pixel-verified) and uses the same event caption as Full+Caption; only the surrounding scene becomes neutral grey (RGB 128). KGR / E4 rise from 13.10 / 9.66 to 16.55 / 13.10 on GPT-5.6 (Det p=0.0064p = 0.0064), from 2.07 / 1.38 to 11.72 / 8.97 on Kimi-K3 (KGR p=0.0001p = 0.0001, E4 p=0.0010p = 0.0010), and from 1.38 / 0.69 to 6.21 / 3.45 on Qwen3.6-27B (KGR p=0.0156p = 0.0156, Det p=2.1×10−7p = 2.1 \times 10^{-7}).
  2. Converging evidence without image modification: A scene-clustered logistic regression over 13,660 model–item observations in the unmodified Full condition, controlling for edit saliency (C1 main figure, C2 secondary figure, C3 background), knowledge family, and domain, shows that scene recognition lowers the odds of detecting a people-family forgery (pooled OR = 0.65, 95% CI [0.51, 0.82], p=0.0004p = 0.0004; negative for 20 of 24 recognizing models; Appendix I, Table 10).
  3. Same caption: Crop+Caption and Full+Caption supply the identical event caption, so the Crop+Caption gain (KGR 1.3×–8.9× that of Full+Caption across all 21 recognizing models) does not come from added text.
  4. Scope: Pad+Caption rules out magnification and coordinate shifts, and the within-Full regression leaves the image unchanged while controlling for coarse saliency. Replacing a cluttered scene with uniform grey can still raise local target salience, and C1/C2/C3 labels do not capture all within-image salience variation. We therefore present these results as converging behavioral evidence that scene recognition biases verification and note the salience confound in the Limitations section.

Full comparison in %. All conditions except Crop+Caption use the 1280 px canvas at native target scale; Crop+Caption uses the native-resolution crop (median 320 px). pp: exact McNemar vs. Full / Full+Caption (FC).

ModelConditionRECOG (fake / real)DetAuthBalKGRE4pp vs. Full / FC
GPT-5.6Full (L1)83.45 / 86.7353.7984.6969.2414.487.59—
GPT-5.6Full+Caption83.45 / 86.7357.2485.7171.4813.109.66—
GPT-5.6Crop+Caption—69.6686.7378.1916.5513.79Det 0.0022 / 0.0096
GPT-5.6Pad30.34 / 38.7864.8369.3967.1115.176.90Det 0.0764 / 0.2000
GPT-5.6Pad+Caption30.34 / 38.7869.6678.5774.1116.5513.10Det 0.0011 / 0.0064; E4 vs. Full 0.0386
Kimi-K3Full (L1)88.28 / 87.7635.8692.8664.363.452.07—
Kimi-K3Full+Caption88.28 / 87.7626.9095.9261.412.071.38—
Kimi-K3Crop+Caption—62.7686.7374.758.286.90Det 4.0×10−84.0 \times 10^{-8} / 9.1×10−139.1 \times 10^{-13}
Kimi-K3Pad15.17 / 15.3115.1794.9055.043.451.38Det 2.8×10−62.8 \times 10^{-6} / 0.0060
Kimi-K3Pad+Caption15.17 / 15.3133.1094.9064.0011.728.97KGR 0.0005 / 0.0001; E4 0.0020 / 0.0010
Qwen3.6-27BFull (L1)73.79 / 77.5540.6956.1248.412.761.38—
Qwen3.6-27BFull+Caption73.79 / 77.5531.7265.3148.521.380.69—
Qwen3.6-27BCrop+Caption—51.7268.3760.055.523.45Det 0.0479 / 1.1×10−41.1 \times 10^{-4}
Qwen3.6-27BPad16.55 / 15.3189.6612.2450.958.280.69Det 1.5×10−191.5 \times 10^{-19} / 2.2×10−242.2 \times 10^{-24}
Qwen3.6-27BPad+Caption16.55 / 15.3156.5566.3361.446.213.45Det 0.0018 / 2.1×10−72.1 \times 10^{-7}; All 0.0012; KGR vs. FC 0.0156
AuthorsOpenReview ↗

response part 3

4. KGR vs. E4 Metric Strictness and Mechanism Interpretation (Response to W4)

We report KGR (E3+) and E4 separately: KGR credits naming the altered entity or fact, and E4 additionally requires explaining why it contradicts the recorded event. Appendix K (Table 15) reports both for all 36 models, and their difference on the same denominator quantifies the credit KGR assigns beyond strict verification.

  • KGR credit without contradiction reasoning: On the same 1,507 forgeries under L1, Gemini-3.8-Flash reaches 55.14% KGR and 37.16% E4. The 17.98-point gap consists of responses that identify the altered entity without stating the specific contradiction, the case your comment describes. Main-text Table 2 will report E4 alongside KGR to separate entity grounding from complete contradiction reasoning.
  • Behavioral scope of "reconstruction from memory": Our evidence is behavioral. On absent-celebrity pairs where a model passes the isolated FACE probe (rejecting the replacement face on a neutral canvas), it still judges the full photograph authentic 63.6% (Kimi-K3), 69.8% (Qwen3.6-27B), and 29.6% (Gemini-3.8-Flash) of the time, and 34 of 126 such REAL responses (27.0%) name the erased person as present. Naming an erased person while judging the photograph authentic is behavioral evidence that familiar-scene priors interfere with verification. Locating the responsible internal circuit would require white-box activation analysis; the strongest models on the benchmark (Gemini-3.8-Flash, GPT-5.6) are proprietary, and mechanistic probing of open-weight models is a separate study. §4.1 accordingly introduces naming the erased person as a behavioral signature of reconstruction from memory.

5. Practical Mitigation Strategies (Response to W5)

The diagnosis suggests keeping scene recognition but separating it from local verification. A training-free two-stage workflow built on this idea improves detection and E4 on all three tested models without a ground-truth caption.

  • Mechanism: Scene recognition supplies the event context needed for verification but also suppresses local checking. Adding the true caption to the full image gives no KGR gain, whereas pairing the same caption with the local crop raises KGR to 1.3×–8.9× across all 21 recognizing models (Table 3).
  • Workflow: Stage 1 queries the full image and keeps the model's own scene description; Stage 2 supplies the local crop with that description and asks the model to verify it. Neither stage uses a ground-truth caption. We use the benchmark's annotated region as the crop; a deployed system would take candidate boxes from an off-the-shelf person or object detector.
  • Effect: On 145 peripheral people-family forgeries and 98 authentic originals, detection rises over Full L1 by +13.80 to +26.90 points on all three models (p≤0.0066p \le 0.0066), and E4 rises on all three (7.59 → 8.97 on GPT-5.6, 2.07 → 7.59 on Kimi-K3 [p=0.0215p = 0.0215], 1.38 → 3.45 on Qwen3.6-27B). KGR rises on Kimi-K3 (3.45 → 12.41) and Qwen3.6-27B (2.76 → 8.28) and is unchanged on GPT-5.6, whose higher detection comes with lower authentic accuracy (84.69 → 69.39), for a 1.7-point gain in balanced accuracy.
ModelConditionDetAuthBalKGRE4Exact McNemar pp vs. Full L1
GPT-5.6Full (L0)28.2895.9262.104.141.38—
GPT-5.6Full (L1)53.7984.6969.2414.487.59—
GPT-5.6Self-Captioned Crop72.4169.3970.9014.488.97Det 0.00014
Kimi-K3Full (L0)17.9395.9256.921.380.69—
Kimi-K3Full (L1)35.8692.8664.363.452.07—
Kimi-K3Self-Captioned Crop49.6689.8069.7312.417.59Det 0.0066; KGR 0.00024; E4 0.0215
Qwen3.6-27BFull (L0)29.6677.5553.602.070.69—
Qwen3.6-27BFull (L1)40.6956.1248.412.761.38—
Qwen3.6-27BSelf-Captioned Crop67.5955.1061.348.283.45Det 1.2×10−61.2 \times 10^{-6}; KGR 0.0215
  • Baseline for future work: E4 stays below 10% on all three models, so the workflow is a training-free baseline rather than a complete solution. It supports decoupling global scene recognition from local verification at inference time, and Mandela-Bench's matched K0 controls and E0–E4 grading provide a standardized testbed for measuring further improvements.
Reviewer_Pt6eOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces Mandela-Bench, a benchmark designed to evaluate whether MLLMs can leverage their internal world knowledge to verify seamlessly edited "canonical images" like iconic historical photographs, film stills, and artworks. Through evaluating 36 models, the authors identify a consistent failure mode termed reconstruction from memory: when a model recognizes a familiar scene, its parametric priors often override the actual visual evidence. This leads models to name individuals who have been erased from a photo or to certify factually incorrect forgeries as authentic. The study demonstrates that even frontier models like GPT-5.6 and Gemini-3.8-Flash suffer from this Mandela effect, where visual recognition biases verification toward the remembered version rather than the observed edit. These findings reveal a critical bottleneck in knowledge-grounded image verification: recognition often triggers a retrieval of memory that blinds the model to contradictory visual details.

Critical strengths
  1. Knowledge forgery is a new evaluation persipective: Prior research has largely relied on detecting generator artifacts physical or logical inconsistencies in pixel level, this paper introduces the concept of knowledge-only forgeries. These images are edited to be visually seamless and intrinsically plausible, ensuring no detectable forensic traces. Consequently, the MLLM must rely entirely on its internal parametric knowledge to verify the image’s authenticity. This framework establishes a novel benchmark for evaluating the knowledge integration capabilities of MLLMs.

  2. The "Mandela Effect" in MLLM is interesting: This paper’s finding reveals an interesting failure mode: reconstruction from memory prevailing over perceptual observation. Experiments demonstrate that once a model recognizes a canonical image, its internal priors exert a powerful bias. Consequently, models may identify a replacement face as a forgery in isolation, yet remain 'blind' to it within the full context, often naming people who have been erased but belong in the original scene. This perceptual-knowledge misalignment exposes a fundamental cognitive bottleneck in how MLLMs process familiar information.

Critical weaknesses
  1. The paper finds that recognition biases verification from memory, but it lacks a quantitative analysis of the triggering threshold. Specifically, it remains unclear at what level of visual degradation (e.g., resolution or noise) the MLLM stops relying on reconstruction from memory and starts performing evidence-based inspection. Defining this boundary between recognition bias and perceptual grounding would provide deeper insights into the mechanism of the observed failures.

  2. Lack of Analysis of Intermediate Reasoning: The paper quantifiably establishes the reconstruction from memory bias but lacks a granular deconstruction of models' internal thought. By focusing primarily on the final verdict, the study leaves it unclear whether failures stem from perceptual neglect or logical integration errors where priors override visual evidence. Analyzing intermediate reasoning steps, especially for frontier models, would better point out the trigger of the Mandela effect and clarify the underlying cognitive bottleneck.

  3. The author mentions using an LLM to polish the text. However, while reading it, I still encountered many instances of unnatural phrasing characteristic of AI. For example: line 86: These items cover six knowledge families—people, objects, places, visible text, event attributes, and works of art—and six image domains ranging from historical photographs to film stills. I hope the authors can polish such discriptions during rebuttal period.

Questions for authors
  1. Please provide an additional experiment showing the dynamic trend of the KGR metric under varying levels of visual interference? Specifically, how does the model's reliance on reconstruction from memory shift as the edited region undergoes increasing intensities of Blur? This would help clarify the precision threshold at which visual evidence begins to be completely discounted in favor of the canonical prior.

  2. In cases scored as E2:correct localization without identification, do models' reasoning chains contain explicit descriptions of perceived visual inconsistencies? If such thinking logs avaliable, could the authors release annotated samples to help the community study the specific triggers of cognitive dissonance when visual perception fails to override memory?

11 comments
AuthorsOpenReview ↗

response part 1

Thank you for the 3/3 ratings on Soundness and Informativeness, and for recognizing the new evaluation perspective of knowledge-only forgeries and the fundamental bottleneck between parametric priors and perceptual observation exposed by the MLLM "Mandela Effect." We report a two-way blur sweep that separates the effects of blurring the edited region and the surrounding scene (W1, Q1), and an audit of 3,810 E2 responses and 126 FACE-pass/Full-REAL responses with released annotated samples (W2, Q2). We also describe our planned manual revision of sentence flow and phrasing throughout the manuscript (W3).

1. Dynamic Trend of KGR and Memory Override Under Varying Blur Intensities (Response to W1 & Q1)

There is no sharp threshold, and the two kinds of blur act in opposite directions. Blurring the edited region (Target-Blur, Q1) removes knowledge-grounded verification from σ=4\sigma = 4 px onward, while models keep naming the erased canonical person. Blurring the surrounding canonical scene (Context-Blur, W1) lowers scene recognition gradually and triples KGR and E4 by σ=16\sigma = 16 px. Both sweeps use σ∈{0,2,4,8,16}\sigma \in \{0, 2, 4, 8, 16\} px.

  • Setup: A deterministic domain-stratified cohort of 60 peripheral people-family (K1) forgeries and 40 matched authentic originals (Cohort 1), plus 50 release-verified absent-celebrity erasures (Cohort 2), evaluated on Kimi-K3 and Qwen3.6-27B with Gaussian blur at native Full resolution:

    1. Target-Blur (Q1): blurs only the edited region, keeping the surrounding canonical scene sharp, to test how degrading local identity evidence affects KGR, E4, and absent-celebrity judgments.
    2. Context-Blur (W1): blurs only the surrounding scene outside the local crop window, keeping the edited region fully sharp at native resolution and pairing it with the event caption, from σ=0\sigma = 0 to complete masking (Pad), to test how degrading the canonical scene affects RECOG, KGR, E4, and Bal.
  • Target-Blur on the edited region (Q1): On Cohort 1, both models retain low but non-zero knowledge grounding under mild blur (σ=2\sigma = 2 px: 3.33% on Kimi-K3 and 1.67% on Qwen3.6-27B). From σ=4\sigma = 4 px onward, KGR and E4 are 0.00% on both models, and all remaining FAKE verdicts are ungrounded E1/E2 remarks about local blur artifacts. On Cohort 2, increasing target blur shifts authenticity verdicts in model-dependent directions: from σ=0\sigma = 0 to 16 px, Kimi-K3's false REAL rate rises from 70.0% to 90.0% (+20.0 pp), whereas Qwen3.6-27B's falls from 78.0% to 50.0% (−28.0 pp). Responses that both name the erased canonical person and judge the image REAL decrease (Kimi-K3 28.0% → 18.0%; Qwen3.6-27B 34.0% → 20.0%), yet both models still name the erased person in 24.0%–28.0% of responses at σ=16\sigma = 16 px. Values in %:

Model / cohortMetricσ\sigma = 02 px4 px8 px16 px
Kimi-K3, Cohort 1 (K1, L1)Det / Auth33.33 / 90.0015.00 / 97.5021.67 / 90.0018.33 / 80.0028.33 / 82.50
KGR / E43.33 / 3.333.33 / 3.330.00 / 0.000.00 / 0.000.00 / 0.00
Ungrounded FAKE (E1+E2)30.0011.6721.6718.3328.33
Qwen3.6-27B, Cohort 1 (K1, L1)Det / Auth41.67 / 47.5076.67 / 40.0083.33 / 27.5086.67 / 15.0086.67 / 12.50
KGR / E43.33 / 3.331.67 / 1.670.00 / 0.000.00 / 0.000.00 / 0.00
Ungrounded FAKE (E1+E2)38.3373.3383.3386.6786.67
Kimi-K3, Cohort 2 (absent-celebrity, L0)Erase Det / REAL verdict30.00 / 70.0020.00 / 80.0018.00 / 82.0020.00 / 80.0010.00 / 90.00
Named erased / named & REAL44.00 / 28.0032.00 / 24.0028.00 / 24.0026.00 / 20.0024.00 / 18.00
Qwen3.6-27B, Cohort 2 (absent-celebrity, L0)Erase Det / REAL verdict20.00 / 78.0024.00 / 76.0070.00 / 30.0064.00 / 36.0050.00 / 50.00
Named erased / named & REAL44.00 / 34.0032.00 / 30.0022.00 / 8.0020.00 / 10.0028.00 / 20.00
  • Context-Blur on the surrounding scene (W1): Blurring the surrounding scene while keeping the edited region sharp produces the opposite trend. From σ=0\sigma = 0 to Pad, RECOG on forgeries drops monotonically from 90.0% to 11.7% on Kimi-K3 and from 68.3% to 16.7% on Qwen3.6-27B. As recognition weakens, verification improves: at σ=4\sigma = 4 px, KGR doubles (3.33% → 6.67% on Kimi-K3; 1.67% → 3.33% on Qwen3.6-27B). At σ=16\sigma = 16 px, KGR and E4 both reach 10.00% (3.0×) on Kimi-K3 and 5.00% (3.0×) on Qwen3.6-27B, while balanced accuracy rises from 61.25% to 64.17% and from 48.75% to 60.00%. Under Pad, Qwen3.6-27B reaches 6.67% KGR (4.0×) and 65.00% balanced accuracy (+16.25 pp), with E4 at 1.67%. The absolute KGR gains are modest and gradual rather than a sharp step function. Values in %; edited region sharp throughout:
AuthorsOpenReview ↗

response part 2

1. Dynamic Trend of KGR and Memory Override Under Varying Blur Intensities (Response to W1 & Q1) (continued)

Surrounding-scene blurKimi-K3 RECOG (fake / real)Kimi-K3 Det / Auth / BalKimi-K3 KGR / E4Qwen3.6-27B RECOG (fake / real)Qwen3.6-27B Det / Auth / BalQwen3.6-27B KGR / E4
σ\sigma = 0 (Full+Caption)90.00 / 92.5030.00 / 92.50 / 61.253.33 / 3.3368.33 / 80.0030.00 / 67.50 / 48.751.67 / 1.67
2 px78.33 / 82.5035.00 / 90.00 / 62.503.33 / 3.3360.00 / 50.0083.33 / 40.00 / 61.673.33 / 3.33
4 px51.67 / 70.0051.67 / 75.00 / 63.336.67 / 6.67 (2.0×)35.00 / 35.0096.67 / 20.00 / 58.333.33 / 1.67 (2.0×)
8 px33.33 / 47.5053.33 / 75.00 / 64.176.67 / 6.67 (2.0×)31.67 / 37.5091.67 / 12.50 / 52.083.33 / 3.33 (2.0×)
16 px15.00 / 35.0038.33 / 90.00 / 64.1710.00 / 10.00 (3.0×)25.00 / 27.5080.00 / 40.00 / 60.005.00 / 5.00 (3.0×)
Pad (complete mask)11.67 / 22.5031.67 / 92.50 / 62.088.33 / 5.00 (2.5×)16.67 / 15.0060.00 / 70.00 / 65.006.67 / 1.67 (4.0×)

2. Granular Deconstruction of Intermediate Reasoning Chains and Release of Annotated Samples (Response to W2 & Q2)

The E2 traces do contain explicit descriptions of visual inconsistencies: 97.7% of inline <think> traces and 98.0% of full responses describe a localized visual anomaly or an identity conflict. On the 126 FACE-pass yet Full = REAL errors, the text separates the two failure types only partly: 2 responses state that the visible person is not the canonical figure and still answer REAL (logical integration failure, e.g., the 1948 Truman case), 45 name the erased person or endorse the scene from memory, and the remaining brief verdicts do not reveal which failure occurred. Both annotated corpora will be released with the benchmark. Counts with shares; the inline column covers 4 Qwen models (NN = 2,114), the full-response column all 7 models (NN = 3,810):

CategoryInline <think> onlyFull saved response
E2 (L0 + L1): explicit localized visual anomaly (B1 + B2)1,722 (81.5%)3,403 (89.3%)
B1: visual anomaly + contextual/historical conflict1,660 (78.5%)2,906 (76.3%)
B2: visual anomaly alone (lighting, texture, proportion, edges)62 (2.9%)497 (13.0%)
B3: contextual/identity conflict alone343 (16.2%)329 (8.6%)
Any localized discrepancy (B1+B2+B3); manual audit 91.3% (42/46)2,065 (97.7%)3,732 (98.0%)
B4: generic suspicion, no local cue49 (2.3%)78 (2.0%)
FACE-pass yet Full = REAL (NN = 126, 81 pairs): (i) explicit prior rationalization—2 (1.6%)
(ii) names erased person in REAL verdict (Kimi-K3 42.9% [15/35], Gemini-3.8 33.3% [8/24])—34 (27.0%)
(iii) holistic scene endorsement (Kimi-K3 17.1% [6/35], Gemini-3.8 12.5% [3/24])—11 (8.7%)
(iv) brief REAL verdict, mechanism not separable from text—79 (62.7%)
  • E2 <think> traces vs. final explanations, and visual vs. contextual cues (Q2): E2 requires localization of the altered figure or region in the final explanation, so we analyzed two scopes separately: (a) the 4 models with full inline <think> logs (Qwen3-VL-235B-A22B-Thinking [428], Qwen3-VL-32B-Thinking [430], Qwen3.6-27B [580], Qwen3.8-27B [676]; NN = 2,114), scored on the <think> text alone; and (b) all 7 models' full responses (NN = 3,810, adding Kimi-K3 [647, 400-char reasoning head + explanation] and Gemini-3.8-Flash [549] and GPT-5.6 [500], final explanations only). Within the <think> traces, 81.5% explicitly describe localized visual anomalies (78.5% B1, 2.9% B2), and another 16.2% state that the visible person does not match the canonical figure (B3), totaling 97.7%. Across all 3,810 full responses, 89.3% describe localized visual anomalies and 8.6% contextual/identity conflict alone, totaling 98.0%. Verbatim E2 examples:
    • Qwen3.8-27B, L1, Thai royal coronation portrait (visual blending/lighting anomalies and identity conflict):

      <think> trace: "Now the face. The face does not look like King Bhumibol Adulyadej (Rama IX) or King Vajiralongkorn (Rama X)... Looking more carefully: the face has a smooth, slightly artificial quality, and the lighting on the face doesn't perfectly match the surrounding image... The mismatch between the face and the rest—skin tone, lighting, the way the crown sits, the ears—suggests manipulation."

AuthorsOpenReview ↗

response part 3

2. Granular Deconstruction of Intermediate Reasoning Chains and Release of Annotated Samples (Response to W2 & Q2) (continued)

  • Qwen3-VL-235B-A22B-Thinking, L0, Marvin Gaye replacing Boromir in The Lord of the Rings (canonical scene memory localizes the replaced face):

    <think> trace: "So, let's see. The image is a meme with text 'ONE DOES NOT SIMPLY WALK INTO MORDOR' which is a famous Lord of the Rings reference. But the person in the image is not from the original movie. The original scene has Boromir, but this is a different person (a meme with a different face). So it's a fake image, edited to replace the original character..."

  • Kimi-K3, L0, 2010 Swedish Royal Wedding carriage scene (saved reasoning head + final explanation):

    Reasoning head: "This looks like it could be the wedding of Crown Princess Victoria of Sweden and Daniel Westling, which took place in 2010. However, the faces in this image look odd - they appear to be different..." Final explanation: "FAKE. This image mimics the famous 2010 wedding carriage ride of Crown Princess Victoria of Sweden and Daniel Westling, but the faces don't match the actual couple — they appear to be AI-generated or digitally swapped in. Telltale signs include the overly smooth, uncanny skin texture on both faces..."

  • Perceptual neglect vs. logical integration failure on FACE-pass / Full = REAL errors (W2): Passing the isolated FACE probe shows that a model can distinguish the real and replacement faces in isolation, not that it noticed the discrepancy in the full image. The 126 audited responses separate as follows:

    • (i) Direct text evidence of logical integration failure (1.6%, 2/126): The model states within its full-image response that the visible person is not the canonical figure, yet still judges the photograph REAL. On the 1948 "Dewey Defeats Truman" photograph, Kimi-K3 writes in its reasoning trace: "However, the famous photo is of Harry Truman holding the newspaper, smiling. This image shows a different man (looks like it could be a radio broadcaster or someone else) holding the paper", and in its final answer: "The man pictured is not Truman — likely a broadcaster or commentator posing with the infamous edition — but the photograph itself shows no clear signs of manipulation or fabrication", returning REAL. The model rationalizes a perceived identity conflict to preserve the authenticity of the canonical scene.
    • (ii) False attribution of the erased person (27.0%, 34/126) and (iii) holistic scene endorsement (8.7%, 11/126): The model names the erased canonical person as present while calling the image REAL (e.g., Qwen3.6-27B: "It shows U.S. President Harry S. Truman holding a copy of the Chicago Daily Tribune..."), or certifies authenticity from global event recognition alone (e.g., Gemini-3.8-Flash on the 2022 G7 summit photo). These responses provide text evidence of memory-based reconstruction, but output text alone cannot determine whether the model neglected to inspect the face or overrode a perceived difference. The remaining 62.7% brief REAL verdicts likewise cannot establish the share of each internal mechanism.
  • Release of annotated thinking logs (Q2): We will release the annotated JSONL corpora (3,810 classified E2 records, 46 manually audited E2 samples with verbatim <think> examples, and 126 annotated FACE-pass / Full-REAL records) with the benchmark repository and add this analysis to §6 and Appendix O.

3. Polishing Phrasing and Improving Prose Fluency (Response to W3)

Several sentences, including Line 86, read stiffly because of compressed lists and punctuation. The manuscript was written by the authors, with an LLM used only for language polishing, as noted in our disclosure statement. We will manually revise the entire manuscript paragraph by paragraph to improve sentence flow and readability.

  • Proposed revision (Line 86, §1 Introduction):

    Original: "These items cover six knowledge families—people, objects, places, visible text, event attributes, and works of art—and six image domains ranging from historical photographs to film stills." Proposed: "The benchmark spans six knowledge families (people, objects, places, visible text, event attributes, and artwork) and six visual domains, including historical photographs, sports, paintings, film stills, political news, and meme templates."

  • We will revise dense lists and transitions across §1–§7 and the appendices in the same way.

Planned Revisions

  1. Blur sweep: §6 and Appendix O add the Target-Blur and Context-Blur tables and curves.
  2. Reasoning chains: §6 and Appendix O add the reasoning-chain breakdown, and the annotated JSONL logs are released with the benchmark.
  3. Prose: The revised manuscript incorporates the manual prose revision described above.
Reviewer_DTnCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Mandela-Bench, a benchmark for testing whether multimodal language models identify factual contradictions in seamless edits of familiar images. The dataset contains 1,359 knowledge-anchored forgeries, 148 edited controls without factual contradictions, and 474 untouched originals. The authors evaluate 36 models using authenticity judgments, evidence-grounded scoring, and probes for factual knowledge, scene recognition, and face identification. Models sometimes name people removed from photographs and accept edited images even when they distinguish replacement faces in isolation. Cropping recognizable scene context while providing correct event information can improve detection.

Critical strengths

The paper addresses a practically relevant failure mode that pixel-level detectors and image–caption consistency benchmarks may miss. Its strongest contribution is the evaluation design rather than a new detection algorithm. Matched anchor-free edits and untouched originals help separate factual inconsistency from generic editing traces, while KGR discourages credit for unsupported “fake” guesses. The combination of text-only, identity, face, and cropping probes provides more insight than a single authenticity score. The comparison across 36 models and detailed appendices also make the work a useful resource.

Critical weaknesses
  • Overclaim: Recognizing a famous Kennedy photo may rely on event-level semantic priors, not image-specific visual memory; cropping changes context/salience, so the mechanism isn’t cleanly isolated.
  • Benchmark conflates knowledge types: Detecting anachronisms vs. replaced figures requires different knowledge; KGR only requires identifying the altered entity, so the stricter E4 results deserve more emphasis.
  • Defense evaluation is limited: No attack-aware defender informed of the face-replacement strategy is tested; class imbalance exists but balanced accuracy partly mitigates it.
Questions for authors
  • Can you disentangle event-level semantic priors from image-specific memory?
  • Would explicitly warning models about seamless face substitution mitigate failures without greatly increasing false positives?
  • Could you emphasize E4 and provide additional results on class-balanced subsets?
4 comments
AuthorsOpenReview ↗

response part 1

Thank you for your review of Mandela-Bench. Our evaluation combines matched anchor-free controls (K0), untouched originals, the E0–E4 evidence ladder, and diagnostic probes across 36 models. This design isolates a practically important failure mode missed by pixel-level detectors and image–caption consistency benchmarks. We address your three questions (Q1–Q3 / W1–W3) below using experimental controls and expanded evaluations.

1. Separating External Event Cues and Surrounding Visual Context, and Clarifying Image vs. Event Priors (Response to W1 & Q1)

Partly. Event-level knowledge supplied as text does not produce the effect: Pad+Caption and Full+Caption share the same event caption, 1280 px canvas, and edited-region position and size, and masking only the surrounding scene improves knowledge-grounded verification on all three tested models. The effect therefore comes from seeing the canonical scene. Whether the seen scene acts through memory of the specific photograph or through the event priors it triggers cannot be separated behaviorally, because a canonical photograph depicts a canonical event; we will state this boundary in the paper.

  1. Pad+Caption vs. Full+Caption: Both conditions supply the same verbatim caption naming the canonical event and date ("Context: this is a crop from a {noun} of {event}, from {date}."); Pad+Caption replaces only the surrounding scene with neutral grey (RGB(128,128,128)). Across GPT-5.6, Kimi-K3, and Qwen3.6-27B, masking the surrounding scene lowers RECOG by 53–73 points and raises both KGR and E4 over Full+Caption on all three models (13.10 / 9.66 → 16.55 / 13.10 on GPT-5.6; 2.07 / 1.38 → 11.72 / 8.97 on Kimi-K3; 1.38 / 0.69 → 6.21 / 3.45 on Qwen3.6-27B). Detection also increases on GPT-5.6 (p=0.0064p = 0.0064) and Qwen3.6-27B (p=2.1×10−7p = 2.1 \times 10^{-7}). Crop+Caption pairs the same caption with the local crop and improves in the same direction on all 21 recognizing models (KGR 1.3×–8.9× its Full+Caption level; pooled KGR +5.62 pp [+3.66, +7.74], Bal +9.88 pp [+7.78, +11.96]). Adding the caption to the full image instead lowers pooled detection by 6.11 pp (95% CI [−9.62, −2.24]).

    Values in % on 145 forged / 98 authentic items; Full, Full+Caption (FC), and Pad+Caption keep the edited region at its native position and size on the 1280 px canvas; Crop+Caption uses the native-resolution crop (median 320 px). pp: exact McNemar vs. Full / FC.

ModelConditionRECOG (fake / real)DetAuthBalKGRE4pp vs. Full / FC
GPT-5.6Full (L1)83.45 / 86.7353.7984.6969.2414.487.59—
GPT-5.6Full+Caption83.45 / 86.7357.2485.7171.4813.109.66—
GPT-5.6Pad+Caption30.34 / 38.7869.6678.5774.1116.5513.10Det 0.0011 / 0.0064; E4 vs. Full 0.0386
GPT-5.6Crop+Caption—69.6686.7378.1916.5513.79Det 0.0022 / 0.0096
Kimi-K3Full (L1)88.28 / 87.7635.8692.8664.363.452.07—
Kimi-K3Full+Caption88.28 / 87.7626.9095.9261.412.071.38—
Kimi-K3Pad+Caption15.17 / 15.3133.1094.9064.0011.728.97KGR 0.0005 / 0.0001; E4 0.0020 / 0.0010
Kimi-K3Crop+Caption—62.7686.7374.758.286.90Det 4.0×10−84.0 \times 10^{-8} / 9.1×10−139.1 \times 10^{-13}
Qwen3.6-27BFull (L1)73.79 / 77.5540.6956.1248.412.761.38—
Qwen3.6-27BFull+Caption73.79 / 77.5531.7265.3148.521.380.69—
Qwen3.6-27BPad+Caption16.55 / 15.3156.5566.3361.446.213.45Det 0.0018 / 2.1×10−72.1 \times 10^{-7}; KGR vs. FC 0.0156
Qwen3.6-27BCrop+Caption—51.7268.3760.055.523.45Det 0.0479 / 1.1×10−41.1 \times 10^{-4}
  1. What the comparison separates, and what remains entangled: Comparing Full+Caption with Pad+Caption separates supplying the event-level fact in text from preserving the surrounding visual scene at matched target scale and position. The surrounding scene can cue both memory of the specific photograph and recognition of the depicted historical event, and a uniform grey surround can also increase local target salience. On absent-celebrity items, models name the erased person in up to 74.2% of L0 responses and 72.7% of L2 responses (e.g., naming Harry Truman on the 1948 "Dewey Defeats Truman" photograph). The unedited surroundings, posture, and props, such as the visible newspaper headline, remain in the image, so a model can arrive at the name either by recalling the iconic photograph or by inferring who should be present from event cues.
AuthorsOpenReview ↗

response part 2

2. Attack-Aware Defender Informed of Seamless Face Substitution (Response to W3 & Q2)

An explicit warning about seamless face substitution/erasure on full images (Attack-Aware Prompt) brings no reliable gain and produces an 88.8% false-alarm rate on Qwen3.6-27B. A two-stage defense (Self-Captioned Crop) that separates global scene description from local crop verification improves detection, balanced accuracy, and E4 on all three tested models without using a ground-truth caption.

  • Protocol: We evaluated two training-free defenders under L1 on the 145 peripheral people-family (K1) forgeries and 98 matched authentic originals:
    1. Attack-Aware Prompt (full image): appends to L1: "Familiar historical scenes may contain seamless face/person substitution or erasure. In your reasoning, first list the key people expected in the canonical scene. Then independently inspect each actually visible person and verify whether their visible facial features match that identity before deciding REAL or FAKE. Do not treat recognition of the scene as evidence that the expected person is present."
    2. Self-Captioned Crop (two-stage): the model first describes the scene from Full, then verifies the local crop together with its self-generated description, without a ground-truth caption. In end-to-end deployment, candidate person/object boxes would come from a standard detector.
  • Result: The full-image warning changes detection by +4.14 pp on GPT-5.6 (p=0.3915p = 0.3915) and −13.10 pp on Kimi-K3; on Qwen3.6-27B, authentic accuracy drops from 56.12% to 11.22% with no KGR gain. Self-Captioned Crop raises detection on all three models (72.41% / 49.66% / 67.59% on GPT-5.6 / Kimi-K3 / Qwen3.6-27B, all p≤0.0066p \le 0.0066 vs. Full L1), balanced accuracy by +1.66 / +5.37 / +12.93 pp, and E4 from 7.59 / 2.07 / 1.38% to 8.97 / 7.59 / 3.45%. KGR rises 3.6× on Kimi-K3 and 3.0× on Qwen3.6-27B. On GPT-5.6, higher detection comes with lower authentic accuracy (84.69% → 69.39%). Values in %; change vs. Full L1 in parentheses; pp: McNemar on Det vs. Full L1:
ModelConditionDetAuthBalKGRE4pp
GPT-5.6Full L0 (default)28.2895.9262.104.141.38<10−9< 10^{-9}
GPT-5.6Full L1 (standard)53.7984.6969.2414.487.59—
GPT-5.6Attack-Aware Prompt57.93 (+4.14)77.55 (−7.14)67.74 (−1.50)13.10 (−1.38)6.90 (−0.69)0.3915 (n.s.)
GPT-5.6Self-Captioned Crop (no GT)72.41 (+18.62)69.39 (−15.30)70.90 (+1.66)14.48 (0.00)8.97 (+1.38)0.00014
Kimi-K3Full L0 (default)17.9395.9256.921.380.69<10−4< 10^{-4}
Kimi-K3Full L1 (standard)35.8692.8664.363.452.07—
Kimi-K3Attack-Aware Prompt22.76 (−13.10)97.96 (+5.10)60.36 (−4.00)6.21 (+2.76)4.14 (+2.07)0.0005 (lower)
Kimi-K3Self-Captioned Crop (no GT)49.66 (+13.80)89.80 (−3.06)69.73 (+5.37)12.41 (+8.96)7.59 (+5.52)0.0066 (2.0×10−62.0 \times 10^{-6} vs. FC)
Qwen3.6-27BFull L0 (default)29.6677.5553.602.070.690.0386
Qwen3.6-27BFull L1 (standard)40.6956.1248.412.761.38—
Qwen3.6-27BAttack-Aware Prompt90.34 (+49.65)11.22 (−44.90)50.78 (+2.37)2.76 (0.00)0.69 (−0.69)<10−20< 10^{-20} (false-alarm collapse)
Qwen3.6-27BSelf-Captioned Crop (no GT)67.59 (+26.90)55.10 (−1.02)61.34 (+12.93)8.28 (+5.52)3.45 (+2.07)1.2×10−61.2 \times 10^{-6}
AuthorsOpenReview ↗

response part 3

3. Emphasizing Stricter E4 Across Knowledge Types and Evaluating Class-Balanced (1:1) Subsets (Response to W2, W3 & Q3)

E4 scores differ sharply between class-level and person-level knowledge. Gemini-3.8-Flash reaches 50–78% E4 on every class-level family (K2–K6) but 12.7% on K1 People; every other model stays below 10% E4 on K1. On a 1:1 scene-matched class-balanced subset, the model tiering is unchanged, with Gemini-3.8-Flash first at 87.4% 1:1 accuracy. We will report E4 and the K0-excluded K1–K6 rates in main-text Table 2 and add both analyses.

  1. E4 across knowledge families and K1 contradiction types (W2 & Q3): Within K1, Gemini-3.8-Flash / GPT-5.6 / Kimi-K3 reach 13.4% / 9.3% / 4.0% E4 on anachronistic lifespans (nn = 322), 7.6% / 13.3% / 13.3% on cross-domain figures (nn = 105), and 12.1% / 5.3% / 0.8% on erased figures (nn = 132). L1 values; cells give KGR (E3+) / E4 in %, with item counts in the header:
ModelAll (1,507)K1–K6 (1,359)K1 People (567)K2 Object (294)K3 Place (89)K4 Text (108)K5 Attr (145)K6 Art (156)
Gemini-3.8-Flash55.1 / 37.260.4 / 41.133.7 / 12.784.7 / 50.369.7 / 56.287.0 / 77.877.9 / 67.671.8 / 68.6
GPT-5.636.0 / 26.339.8 / 29.118.7 / 9.368.0 / 45.239.3 / 36.069.4 / 63.950.3 / 43.433.3 / 29.5
GLM-5.3-Flash34.8 / 29.638.3 / 32.811.1 / 6.267.7 / 58.847.2 / 42.768.5 / 64.849.7 / 44.844.9 / 41.7
Kimi-K334.1 / 29.737.6 / 32.78.8 / 5.163.3 / 52.429.2 / 27.081.5 / 77.860.7 / 57.946.8 / 44.9
Kimi-K2.632.0 / 25.635.4 / 28.311.3 / 6.360.9 / 45.925.8 / 24.772.2 / 66.755.2 / 46.936.5 / 33.3
Qwen3.5-122B-A10B31.5 / 21.834.7 / 24.113.8 / 6.260.9 / 40.127.0 / 23.663.9 / 55.647.6 / 36.634.0 / 26.3
Qwen3-VL-235B-A22B26.9 / 16.929.7 / 18.811.5 / 4.661.6 / 39.136.0 / 29.246.3 / 37.033.1 / 19.317.9 / 12.8
Qwen3.6-27B26.1 / 15.728.7 / 17.47.8 / 2.657.1 / 29.622.5 / 13.562.0 / 52.831.0 / 20.729.5 / 22.4
  1. 1:1 scene-matched class-balanced subset (W3 & Q3): We evaluated a strict 1:1 scene-matched subset (nn = 368 forged : 368 authentic from the same scenes, 1,000 Monte Carlo draws, [2.5%, 97.5%] intervals). Under L1, Gemini-3.8-Flash ranks first on 1:1 accuracy, F1, KGR, and E4 (87.4% accuracy / 87.3 F1 / 49.9% E4), followed by Kimi-K3, GPT-5.6, Kimi-K2.6, and GLM-5.3-Flash by 1:1 accuracy:
ModelL0 Acc [2.5–97.5%]L0 F1L0 KGR / E4L1 Acc [2.5–97.5%]L1 F1L1 Det / AuthL1 KGR [2.5–97.5%]L1 E4 [2.5–97.5%]
Gemini-3.8-Flash83.9 [82.5, 85.2]82.055.7 / 37.887.4 [86.1, 88.6]87.386.5 / 88.367.8 [64.4, 70.9]49.9 [47.0, 53.0]
Kimi-K374.8 [73.4, 76.5]68.133.5 / 24.782.6 [81.1, 84.1]80.672.2 / 92.948.0 [44.8, 51.4]42.8 [39.7, 45.9]
GPT-5.670.1 [68.3, 71.7]59.423.6 / 11.979.9 [78.3, 81.4]77.368.7 / 91.046.0 [42.7, 49.2]35.8 [33.2, 38.9]
Kimi-K2.670.8 [69.2, 72.4]61.229.3 / 21.278.2 [76.6, 79.8]75.165.7 / 90.844.4 [41.6, 47.6]36.7 [34.0, 39.4]
GLM-5.3-Flash75.0 [73.4, 76.6]69.736.8 / 28.076.5 [75.0, 78.0]74.167.3 / 85.645.2 [42.1, 48.1]38.2 [35.3, 41.0]
Qwen3.5-122B-A10B67.9 [66.3, 69.7]61.530.3 / 19.371.4 [69.7, 73.0]68.963.4 / 79.341.3 [38.0, 44.3]29.5 [26.6, 32.3]
Qwen3-VL-235B-A22B65.7 [64.0, 67.3]52.922.9 / 15.069.3 [67.9, 70.8]68.466.6 / 72.034.2 [31.0, 37.2]20.2 [17.7, 22.8]
Qwen3.6-27B64.8 [63.2, 66.4]58.726.9 / 16.566.7 [65.2, 68.3]67.268.3 / 65.235.3 [32.6, 38.0]22.8 [20.4, 25.3]

Planned Revisions

  1. Visual context vs. event cues: Table 3 adds the scale-matched Pad / Pad+Caption control, with discussion in §5. The Abstract, §1, §5, and §7 describe the effect isolated by these experiments: visually recognizing the canonical scene/event context relative to the same text caption. The Limitations section notes that photograph-specific visual memory and visually triggered event-level priors are not fully disentangled.
  2. Attack-aware defender: §6 adds a comparison of attack-aware prompting and two-stage self-captioned crop verification, with the full table in Appendix J.
  3. E4 and class balance: Table 2 reports E4 alongside KGR. §5 and Appendix Table 15 add E4 results by knowledge family and K1 contradiction type. Appendix K adds results on the 1:1 scene-matched class-balanced subset.

Thank you once again for the time and effort you have devoted to reviewing our paper. Your thoughtful comments have been valuable in helping us refine and strengthen our work. Should you have any additional questions or feel that any part of our response needs further clarification, please do not hesitate to let us know. We would be glad to provide further explanations and discuss any remaining points.

Reviewer_s6SeOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Mandela-Bench, a benchmark for testing whether multimodal models can detect seamless manipulations of canonical images using their internal world knowledge rather than low-level artifacts or external references. The authors find that recognition can actually hurt verification: models often reconstruct the remembered canonical image, even naming erased people or accepting manipulated photographs despite being able to detect the altered face in isolation.

Critical strengths
  1. The benchmark isolates an important failure mode largely missed by existing forgery benchmarks: factual inconsistencies where the pixels themselves provide few forensic cues.
  2. The experiments and metrics are well designed. Rather than reporting only fake/real accuracy, the paper separates recognition, factual knowledge, visual perception, and final verification using multiple probes and de-recognition experiments.
Critical weaknesses
  1. The paper argues that models rely on their memory of famous images instead of carefully checking what is actually shown. However, the cropping experiment changes several things at once, not just image recognition. There is a change in visual composition, visual attention, etc. The results show that recognition affects performance, but they do not fully prove exactly why this happens inside the model.
  2. The benchmark construction uses a single image editor, also named-entities are selected based on whether one probe model can recognize the portraits. This limits generalization.
  3. The main de-recognition experiment is limited to 145 people-related forgeries with peripheral edits. It is therefore unclear whether the same effect generalizes to other knowledge categories or to edits located in more central parts of the image.
  4. The main KGR metric does not always require the model to explain the factual contradiction that makes the canonical image incorrect. Since identifying the edited entity may be easier than demonstrating knowledge-based verification, the stricter E4 metric may better capture the paper’s central goal.
Questions for authors

Look at weaknesses. Please address my weaknesses in the rebuttal.

9 comments
AuthorsOpenReview ↗

response part 1

Thank you for the 3/3 ratings on Soundness and Presentation, and for recognizing the benchmark's focus on an under-studied failure mode and its use of multiple probes to separate scene recognition, factual knowledge, visual perception, and verification. We address W1–W4 using evidence from the submitted manuscript and targeted new experiments.

1. Separating Recognition from the Other Changes Cropping Introduces (Response to W1)

Two analyses separate recognition from the other changes cropping introduces, and both show that recognizing the scene lowers verification. Pad+Caption holds the event caption, the full 1280 px image dimensions, and the edited region's pixel size and position fixed, replacing only the surrounding scene with neutral grey; all three tested models then achieve higher entity-or-fact grounding (KGR) and contradiction reasoning (E4) than under Full+Caption. Within unmodified Full images, recognizing the scene still lowers detection after controlling for edit saliency.

On the 145 peripheral forgeries and 98 authentic counterparts, the grey surround lowers scene recognition (RECOG) by 53–73 points. Values in %; Full, Full+Caption, and Pad+Caption keep the edited region at its native position and size on the 1280 px canvas; Crop+Caption uses the native-resolution crop (median 320 px). pp: exact McNemar vs. Full / Full+Caption (FC).

ModelConditionRECOG (fake / real)DetAuthBalKGRE4pp vs. Full / FC
GPT-5.6Full (L1)83.45 / 86.7353.7984.6969.2414.487.59—
GPT-5.6Full+Caption83.45 / 86.7357.2485.7171.4813.109.66—
GPT-5.6Pad+Caption30.34 / 38.7869.6678.5774.1116.5513.10Det 0.0011 / 0.0064; E4 vs. Full 0.0386
GPT-5.6Crop+Caption—69.6686.7378.1916.5513.79Det 0.0022 / 0.0096
Kimi-K3Full (L1)88.28 / 87.7635.8692.8664.363.452.07—
Kimi-K3Full+Caption88.28 / 87.7626.9095.9261.412.071.38—
Kimi-K3Pad+Caption15.17 / 15.3133.1094.9064.0011.728.97KGR 0.0005 / 0.0001; E4 0.0020 / 0.0010
Kimi-K3Crop+Caption—62.7686.7374.758.286.90Det 4.0×10−84.0 \times 10^{-8} / 9.1×10−139.1 \times 10^{-13}
Qwen3.6-27BFull (L1)73.79 / 77.5540.6956.1248.412.761.38—
Qwen3.6-27BFull+Caption73.79 / 77.5531.7265.3148.521.380.69—
Qwen3.6-27BPad+Caption16.55 / 15.3156.5566.3361.446.213.45Det 0.0018 / 2.1×10−72.1 \times 10^{-7}; KGR vs. FC 0.0156
Qwen3.6-27BCrop+Caption—51.7268.3760.055.523.45Det 0.0479 / 1.1×10−41.1 \times 10^{-4}

On uncropped, unmasked full images (Full, Appendix I Table 10), a scene-clustered logistic regression across 13,660 model–item observations, controlling for edit saliency (C1/C2/C3), knowledge family, and domain, likewise shows lower odds of detecting a K1 forgery when the canonical scene is recognized (OR = 0.65, 95% CI [0.51, 0.82], p=0.0004p = 0.0004). Uniform grey can still make the target region more salient than a cluttered background, and these behavioral controls do not isolate internal mechanisms; §4.1 accordingly introduces naming the erased person as a behavioral signature of "reconstruction from memory".

2. Single Editor and Portrait Gate: Cross-Editor and Reverse-Selection Tests (Response to W2)

Changing either construction choice leaves the main patterns intact. A paired subset regenerated with a second editor (Qwen-Image-2.1) reproduces the four core regularities within 11.5 points per model, and on a new K1 set selected by Qwen3.6-27B instead of Kimi-K3, the model ranking is unchanged and Qwen3.6-27B reaches only 13.0% KGR.

  1. Single editor
    • Cross-editor test (GPT Image 2 vs. Qwen-Image-2.1): We pre-specified 155 edit tasks on 100 seed images, spanning K1–K6, matched K0 controls, and same-seed absent-celebrity vs. celebrity-swap pairs, regenerated them with the open-weight Qwen-Image-2.1, and applied the same QC pipeline; 96 edits on 70 seeds passed. For Gemini-3.8-Flash, Kimi-K3, and GPT-5.6, the main regularities hold under both editors: class-level K2–K6 KGR (54–77%) stays far above person-level K1 KGR (11–44%), K0 KGR is at most 8.7%, and swaps are detected 27–50 points more often than erasures. Across these four metrics, per-model values differ by at most 11.5 points between editors. The 9-scene Crop+Caption subset has intervals too wide for a reliable editor comparison. Each cell gives GPT Image 2 / Qwen-Image-2.1 with 95% CIs:
AuthorsOpenReview ↗

response part 2

2. Single Editor and Portrait Gate: Cross-Editor and Reverse-Selection Tests (Response to W2) (continued)

Metric (paired nn)Gemini-3.8-FlashKimi-K3GPT-5.6
K1 People KGR (L1, %; nn = 27, Kimi 26)44.4 [25.5, 64.7] / 44.4 [25.5, 64.7]19.2 [6.6, 39.4] / 15.4 [4.4, 34.9]11.1 [2.4, 29.2] / 18.5 [6.3, 38.1]
K2–K6 class-level KGR (L1, %; nn = 35)77.1 [59.9, 89.6] / 77.1 [59.9, 89.6]60.0 [42.1, 76.1] / 54.3 [36.6, 71.2]57.1 [39.4, 73.7] / 68.6 [50.7, 83.1]
K0 control KGR (L1, %; nn = 23)8.7 [1.1, 28.0] / 8.7 [1.1, 28.0]0.0 [0.0, 14.8] / 4.3 [0.1, 21.9]0.0 [0.0, 14.8] / 8.7 [1.1, 28.0]
Swap − erase detection gap (L0, pp; nn = 11, Kimi 10)+27.3 [+0.0, +54.5] / +27.3 [−9.1, +63.6]+40.0 [+10.0, +70.0] / +50.0 [+20.0, +80.0]+45.5 [+18.2, +72.7] / +45.5 [+18.2, +72.7]
Recognition penalty Bal(Crop+Cap) − Bal(Full) (pp; nn = 9)−11.1 [−33.3, +11.1] / −22.2 [−44.4, +0.0]+11.1 [−11.1, +33.3] / −22.2 [−55.6, +5.6]+0.0 [−16.7, +16.7] / +11.1 [+0.0, +27.8]
  • Why the benchmark keeps one high-fidelity editor: The benchmark measures knowledge-based verification when pixel cues are minimal, so a less faithful editor would let models flag edits from artifacts alone. On the paired subset, both pixel detectors find Qwen-Image-2.1 edits easier to detect (NPR AUC 0.568 → 0.658, UniversalFakeDetect AUC 0.527 → 0.635). One editor also gives K1–K6 forgeries and their K0 controls the same editing traces, the control Reviewers 4yrk and 7net credit. CertainlyUncertain (ICLR 2025) and Pixels Versus Priors (EMNLP 2025) likewise build their counterfactuals with one fixed editing procedure. Full-scale benchmarking across editors remains future work.
  1. Portrait gate
    • Scope: During construction, Kimi-K3 checked whether each candidate's unedited Wikimedia portrait was recognizable in isolation, and only accepted candidates entered K1. The gate covers the 435 named-person insertions (28.9% of the 1,507 forgeries); the other 1,072 forged items (132 absent-celebrity erasures, 792 K2–K6 items, 148 K0 controls) do not use it. Model-based screening during construction is also used by Pixels Versus Priors, VisMin (NeurIPS 2024), and SpuriVerse (NeurIPS 2025).
    • Test: If the choice of gate model shaped the results, the model whose recognition defines a subset should verify that subset best. It does not. We split accepted items by whether Qwen3.6-27B also recognized the portrait (Both-recognized, nn = 208; Kimi-only, nn = 227). On the Kimi-only items, Kimi-K3 reaches 5.3% KGR, below Gemini-3.8-Flash (25.6%), GPT-5.6 (10.1%), Qwen3.5-122B (9.7%), GLM-5.3-Flash (7.0%), and Kimi-K2.6 (6.2%). For the reverse case, we audited all 1,247 candidate QIDs probed during construction (294 Both-recognized, 391 Kimi-only, 54 Qwen-only, 508 Neither), took all 43 Qwen-only candidates with pre-existing historical scene specifications, froze the list before generation, and rendered them with GPT Image 2 under the Appendix C QC pipeline, yielding 24 Qwen-only K1 forgeries (NPR AUC 0.655, UniversalFakeDetect AUC 0.530). Although Qwen3.6-27B recognized these reference portraits and Kimi-K3 did not, Qwen3.6-27B grounds only 13.0% (3/23) of the edited scenes, and the ordering on Det and KGR (Gemini-3.8-Flash > Kimi-K3 > Qwen3.6-27B) is identical across all three strata:
Model (L1; KGR with 95% Wilson CI)Both-recognized (nn = 208): Det / KGR / E4Kimi-only (nn = 227): Det / KGR / E4Qwen-only (new, nn = 24): Det / KGR / E4
Gemini-3.8-Flash87.5 / 54.3 [47.5, 61.0] / 17.380.2 / 25.6 [20.3, 31.6] / 8.879.2 / 45.8 [27.9, 64.9] (11/24) / 8.3 (2/24)
Kimi-K367.8 / 17.3 [12.8, 23.0] / 9.669.6 / 5.3 [3.1, 9.0] / 3.554.5 / 18.2 [7.3, 38.5] (4/22) / 13.6 (3/22)
Qwen3.6-27B65.4 / 14.9 [10.7, 20.4] / 5.363.0 / 1.3 [0.5, 3.8] / 0.452.2 / 13.0 [4.5, 32.1] (3/23) / 8.7 (2/23)

The strata contain different scenes, so absolute rates are not comparable across columns; the test is the ranking within each column.

3. Generalization of De-Recognition to Other Knowledge Categories (K2–K5) and Central Edits (C1) (Response to W3)

De-recognition generalizes to non-face knowledge families (K2 Objects, K3 Places, K5 Event Attributes) and to central primary subjects (C1). The submitted manuscript reports Full, Mirror, and Crop across all 902 peripheral forgeries (Appendix K Table 19, with Appendix H Table 7), the four-condition K1–K5 breakdown (Appendix J Table 12), and the full C1/C2/C3 saliency breakdown (Appendix K Table 14). Below, we extend the K1–K5 Crop+Caption comparison to eight models and report a new 140-item central-figure (C1) experiment with Crop+Caption and Pad+Caption.

AuthorsOpenReview ↗

response part 3

3. Generalization of De-Recognition to Other Knowledge Categories (K2–K5) and Central Edits (C1) (Response to W3) (continued)

  1. Other knowledge categories (K1–K5, 902 peripheral forgeries): Appendix Table 12 reports the four conditions by family for Gemini-3.8-Flash and GPT-5.6; we extend it to six open-weight models. For all eight models, Crop+Caption raises KGR over Full on K1 People, K2 Objects (+6.8 to +19.7 points), K3 Places (+9.0 to +34.8), and K5 Event Attributes (+1.4 to +15.9), and detection also rises on K1, K2, and K3. Comparing Full+Caption with Crop+Caption separates person-specific identity (K1) from class-level knowledge (K2–K5): on K1, Full+Caption does not improve KGR and lowers detection for 15 of 21 models, whereas on K4 Text and K5 Attributes it improves detection (+13.9 and +13.1 points on GPT-5.6, p≤0.0026p \le 0.0026) and Crop+Caption raises KGR further. Each cell gives Det (KGR), Full → Crop+Caption, L1:
ModelK1 People (145)K2 Objects (294)K3 Places (89)K4 Text (108)K5 Attributes (145)
Gemini-3.8-Flash73.1→86.2 (35.2→44.8)96.3→100.0 (84.7→91.5)85.4→97.8 (69.7→88.8)90.7→90.7 (87.0→84.3)86.2→95.9 (77.9→87.6)
GPT-5.653.8→69.7 (14.5→16.6)89.1→94.6 (68.0→74.8)60.7→79.8 (39.3→48.3)75.0→81.5 (69.4→70.4)59.3→73.1 (50.3→66.2)
GLM-5.3-Flash52.4→59.3 (7.6→10.3)82.3→93.5 (67.7→79.9)73.0→86.5 (47.2→67.4)74.1→75.0 (68.5→70.4)58.6→63.4 (49.7→58.6)
Kimi-K335.9→62.8 (3.4→8.3)81.0→96.3 (63.3→80.6)61.8→89.9 (29.2→61.8)86.1→85.2 (81.5→83.3)66.9→75.9 (60.7→62.1)
Kimi-K2.628.3→63.4 (4.1→15.2)80.3→96.3 (60.9→80.6)53.9→93.3 (25.8→60.7)79.6→84.3 (72.2→77.8)66.9→71.0 (55.2→65.5)
Qwen3.5-122B-A10B31.7→49.7 (3.4→10.3)80.3→94.9 (60.9→72.8)65.2→86.5 (27.0→52.8)70.4→78.7 (63.9→75.0)61.4→69.7 (47.6→53.1)
Qwen3-VL-235B-A22B48.3→55.9 (4.1→10.3)85.7→95.9 (61.6→72.8)74.2→91.0 (36.0→51.7)54.6→75.0 (46.3→66.7)55.9→53.1 (33.1→40.0)
Qwen3.6-27B40.7→51.7 (2.8→5.5)83.3→91.8 (57.1→63.9)66.3→84.3 (22.5→40.4)66.7→67.6 (62.0→69.4)59.3→49.7 (31.0→40.7)
  1. Central (C1) edits:
    • Existing evidence: The 132 absent-celebrity items and 119 FACE pairs already replace the central protagonist (e.g., JFK, Truman, Maradona), and models that tell the faces apart in isolation still judge many of the full photographs authentic (Appendix H Table 8). Appendix K (Table 14) reports L1 detection and KGR for all 605 central (C1), 72 secondary (C2), and 830 background (C3) forgeries; on central K1 edits (nn = 422), L1 KGR ranges from 9.5% (Qwen3.6-27B) to 33.2% (Gemini-3.8-Flash).
    • New central-figure experiment (80 central K1 named insertions + 60 matched authentic images): Removing the surrounding scene with Crop+Caption or scale-matched Pad+Caption lowers RECOG on forgeries by 38.8–45.0 points (p<10−8p < 10^{-8}). Relative to Full, KGR rises by +5.00 to +6.25 points and E4 by 2.4×–4.5× (Kimi-K3 6.25% → 16.25%, p=0.0386p = 0.0386; Qwen3.6-27B 2.50% → 11.25%, p=0.0156p = 0.0156). Qwen3.6-27B balanced accuracy rises by +10.20 points under Crop+Caption and +13.54 points under Pad+Caption (p=0.0026p = 0.0026). Values in %; change vs. Full in parentheses; pp: exact McNemar vs. Full:
ModelConditionRECOG (fake / real)DetAuthBalKGRE4pp vs. Full
Kimi-K3Full (L1)50.00 / 86.6781.2588.3384.7910.006.25—
Kimi-K3Full+Caption50.00 / 86.6771.25 (−10.00)90.0080.62 (−4.17)12.5012.50Det 0.115
Kimi-K3Crop+Caption7.50 / 51.6775.0090.0082.5016.25 (+6.25)16.25 (+10.00)E4 0.0386
Kimi-K3Pad+Caption11.25 / 46.6777.5093.3385.42 (+0.63)16.25 (+6.25)15.00 (+8.75)E4 0.0923
Qwen3.6-27BFull (L1)52.50 / 65.0073.7550.0061.886.252.50—
Qwen3.6-27BFull+Caption52.50 / 65.0078.75 (+5.00)48.3363.54 (+1.66)8.758.75Det 0.503
Qwen3.6-27BCrop+Caption7.50 / 28.3387.50 (+13.75)56.6772.08 (+10.20)11.25 (+5.00)11.25 (+8.75)Det 0.0347; E4 0.0156; All 0.0357
Qwen3.6-27BPad+Caption10.00 / 33.3387.50 (+13.75)63.3375.42 (+13.54)11.25 (+5.00)8.75 (+6.25)Det 0.0192; All 0.0026 (0.0015 vs. FC)
Reviewer_7netOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Modern image editors can change a famous photo so seamlessly that no pixel-level trace remains; the only evidence is a fact. The question the authors ask is: when an MLLM recognizes a famous image, does it use what it knows to check whether the image is real or not? To this end, they propose a benchmark, Mandela-Bench, with 1,359 forgeries. The findings conclude that only one model exceeds 50 percent. Asking models to verify raises fake verdicts more than the grounded ones. Adding a true caption to the full image does not help.

Critical strengths
  1. A timely, well-controlled benchmark design that tests something new. Forgeries are paired with controls made the same way and with untouched originals from the same source. That neutralizes the two obvious shortcuts: editor artifacts and “famous means real.”

  2. Diagnostic probes plus breadth produce striking, well-documented phenomena. Separate probes for knowing the fact, seeing the face, recognizing the scene, and verifying let the paper argue where failures arise. The judge calibration confusion matrix is reported openly.

Critical weaknesses
  1. The central causal claim is not separated from a simpler visual explanation. Cropping also zooms: images are re-encoded at 1280 px on the longest side, so a crop enlarges the edited region relative to the frame. The authors also admit cropping can bias the model toward fake.

  2. The face probe has the same problem. The mirror control does not address scale.

  3. Benchmark validity and measurement are not independently established. “Knowledge-only” is asserted, not shown. Models being evaluated are also involved in building and grading the benchmark, which introduces bias.

  4. The KGR metric is shaped oddly. It includes the 148 control items in the denominator, which can’t be grounded, so the ceiling is about 90 percent. There is no human baseline.

Questions for authors
  1. Can authors add a scale-matched control, such as a zoomed crop that keeps the composition, higher-resolution full images, or the edited region on a neutral canvas, to separate recognition from resolution?

  2. Please report a human baseline, both with and without the relevant fact, to validate the “knowledge-only” property.

9 comments
AuthorsOpenReview ↗

response part 1

Thank you for the review. As you note, pairing knowledge-anchored forgeries with same-editor K0 controls and untouched originals "neutralizes the two obvious shortcuts: editor artifacts and 'famous means real'", and the multi-probe suite and open judge calibration yield "striking, well-documented phenomena." Below we address the scale controls for Crop and FACE (W1, W2, Q1), measurement independence and the human baseline (W3, Q2), the KGR ceiling (W4), and release ethics, drawing on pixel audits, the requested scale-matched neutral-canvas control, and a new two-stage human evaluation.

1. Separating Scene Recognition from Resolution/Scale (Pad+Caption Neutral-Canvas Control, Response to W1, W2 & Q1)

Our pipeline applies the 1280 px normalization of Appendix C once, to the full image, before cropping (the revised Appendix C states this order), and Appendix N specifies a native-resolution FACE probe. Our pixel audit confirms that neither Crop (median 320 × 320 px) nor FACE (median 227 × 217 px on a neutral canvas; 0/238 at 1280 px) is upscaled to 1280 px after cropping. To also remove changes in relative frame share, we ran the requested scale-matched control: the edited region at native resolution on a full-size 1280 px neutral canvas (Pad, Pad+Caption). It reduces scene recognition by 53–73 points, raises KGR and E4 on all three tested models, and raises Det and Bal on GPT-5.6 and Qwen3.6-27B.

  1. Native pixel audit:
    • All 902 Crop images match the pixel dimensions of the crop box cut from the 1280 px Full image: median 320 × 320 px (IQR 320–372 × 320–375 px, min 175 × 173 px); the edited region covers a median 9.2% of the Full frame.
    • All 238 FACE images (119 original + 119 edited faces) are cut at 1:1 native resolution (median 227 × 217 px, IQR 160–327 × 155–298 px) and placed on a neutral canvas with a 40 px border (median 307 × 297 px); none has a 1280 px dimension (Appendix N).
  2. Scale-matched neutral-canvas control (Q1): Without upscaling, a 320 × 320 px crop still has a smaller frame than a 1280 × 1054 px full image, which could raise the edited region's relative frame share if the vision encoder resizes inputs. Pad removes this factor on all 145 peripheral K1 forgeries and 98 matched authentic originals: it keeps the Full canvas (1280 px longest side, lossless PNG), keeps every pixel inside the Crop box at its native size and coordinates (243/243 pixel-identity verified), and fills the outside with neutral grey (RGB 128), so resolution and frame share are identical to Full. Masking the scene drops RECOG by 53–73 points (83.45% → 30.34% on GPT-5.6, 88.28% → 15.17% on Kimi-K3, 73.79% → 16.55% on Qwen3.6-27B), and Pad+Caption improves KGR and E4 over both Full and Full+Caption on all three models:
    • GPT-5.6: Det rises to 69.66%, equal to Crop+Caption; Bal to 74.11% (+4.87 points); KGR to 16.55%, equal to Crop+Caption; and E4 to 13.10% (7.59% on Full).
    • Kimi-K3: Det stays below Crop+Caption (33.10% vs. 62.76%, with 94.90% authentic accuracy), but KGR rises 3.4× to 11.72% and E4 4.3× to 8.97%, both above Crop+Caption (8.28% / 6.90%).
    • Qwen3.6-27B: Det rises from 40.69% (Full) and 31.72% (Full+Caption) to 56.55%, Bal by +13.03 points to 61.44%, and KGR / E4 to 6.21% / 3.45%.

All values in %, nn = 145 forged / 98 authentic. Full, Full+Caption, and Pad+Caption use the 1280 px canvas at native target scale; Crop+Caption uses the native-resolution crop (about 320 px). pp: exact McNemar vs. Full / Full+Caption (FC); "All" is accuracy over all 243 images.

ModelConditionRECOG (fake / real)DetAuthBalKGRE4pp vs. Full / FC
GPT-5.6Full (L1)83.45 / 86.7353.7984.6969.2414.487.59—
GPT-5.6Full+Caption83.45 / 86.7357.2485.7171.4813.109.66Det 0.4421 / —
GPT-5.6Pad+Caption30.34 / 38.7869.6678.5774.1116.5513.10Det 0.0011 / 0.0064; E4 vs. Full 0.0386
GPT-5.6Crop+Caption42.10 (all)69.6686.7378.1916.5513.79Det 0.0022 / 0.0096
Kimi-K3Full (L1)88.28 / 87.7635.8692.8664.363.452.07—
Kimi-K3Full+Caption88.28 / 87.7626.9095.9261.412.071.38Det 0.0192 / —
Kimi-K3Pad+Caption15.17 / 15.3133.1094.9064.0011.728.97KGR 0.0005 / 0.0001; E4 0.0020 / 0.0010
Kimi-K3Crop+Caption31.30 (all)62.7686.7374.758.286.90Det 4.0×10−84.0 \times 10^{-8} / 9.1×10−139.1 \times 10^{-13}
Qwen3.6-27BFull (L1)73.79 / 77.5540.6956.1248.412.761.38—
Qwen3.6-27BFull+Caption73.79 / 77.5531.7265.3148.521.380.69Det 0.0410 / —
Qwen3.6-27BPad+Caption16.55 / 15.3156.5566.3361.446.213.45Det 0.0018 / 2.1×10−72.1 \times 10^{-7}; All 0.0012; KGR vs. FC 0.0156
AuthorsOpenReview ↗

response part 2

1. Separating Scene Recognition from Resolution/Scale (Pad+Caption Neutral-Canvas Control, Response to W1, W2 & Q1) (continued)

ModelConditionRECOG (fake / real)DetAuthBalKGRE4pp vs. Full / FC
Qwen3.6-27BCrop+Caption42.20 (all)51.7268.3760.055.523.45Det 0.0479 / 1.1×10−41.1 \times 10^{-4}
  1. Two scale-invariant controls already in the manuscript:
    • Non-recognizing models (RECOG < 50%, Figure 4b): 11 models receive the same Crop images, with identical scale and frame-share changes, but do not recognize the canonical scenes under Full. Their median recognition penalty is +0.4 points and their KGR stays near zero.
    • Uncropped within-Full regression (Appendix I, Table 10): over 13,660 observations at a constant 1280 px scale, controlling for edit saliency (C1/C2/C3), family, and domain, recognizing the scene lowers K1 detection odds (OR = 0.65, 95% CI [0.51, 0.82], p=0.0004p = 0.0004).

2. Independence of Benchmark Construction/Grading and Human Baseline With vs. Without the Fact (Response to W3 & Q2)

Of the 36 evaluated models, 33 had no role in benchmark construction or grading, and binary REAL/FAKE verdicts are parsed by deterministic rules without an LLM judge (§4.3). The explanation judge agrees with the 206 blind human labels in Appendix F (Table 4, κw=0.893\kappa_w = 0.893) and with an independent second judge (Kimi-K3 across 4,159 rows, κw=0.946–0.975\kappa_w = 0.946–0.975), with no self-preference inflation. Beyond the 148 matched K0 controls and the construction-time human audits (Appendix C), we ran the requested two-stage human evaluation (without vs. with the relevant fact) on 60 distinct canonical scenes.

  1. Independence of construction and grading:
    • External models: 33 of 36 models had no role in candidate nomination, portrait filtering, or grading, including Gemini-3.8-Flash (1st), GPT-5.6 (2nd), GLM-5.3-Flash (3rd), Kimi-K2.6 (5th), Qwen3.5-122B (6th), Qwen3-VL-235B (7th), Gemma-4-31B, MiniMax-M3, and InternVL3.5-30B.
    • Kimi-K3's construction role: Candidate facts (birth/death dates, object inception years, event participants) were verified deterministically against Wikidata and curated fact tables, face similarity was gated by InsightFace ArcFace, and every release item was human-inspected. If construction involvement favored a model, Kimi-K3 would lead on the items it helped select; instead it ranks 4th overall (34.1% L1 KGR) and 7th on K1 People (8.8% L1 KGR, 5.1% E4). Splitting the 435 named K1 items by whether the reference portrait was recognized by both Kimi-K3 and Qwen3.6-27B (n = 208) or by Kimi-K3 only (n = 227), Kimi-K3 reaches 5.3% KGR on its own subset, behind Gemini-3.8-Flash (25.6%), GPT-5.6 (10.1%), Qwen3.5-122B (9.7%), GLM-5.3-Flash (7.0%), and Kimi-K2.6 (6.2%).
    • Judge calibration: The explanation judge, Qwen3.8-27B, ranks 12th of 15 main models (22.3% L1 KGR, 13.3% L0 KGR, 1.1% L0 K1 KGR). On the 206 blind human-graded answers (Appendix F, Table 4), it is slightly more conservative than humans (KGR 36.9% [76/206] vs. 39.8% [82/206]; E4 18.4% [38/206] vs. 22.8% [47/206]). Across 4,159 release rows, it assigns lower KGR than the independent Kimi-K3 judge in every paired comparison below, and model rankings are identical:
Qwen3.8-27B vs.ScopePaired nnExact E0–E4Within ±1KGR (E3+) agreeCohen's κ\kappaWeighted κw\kappa_wKGR / E4: Qwen3.8 vs. other
Kimi-K3 judgeGPT-5.6 (L0 / L1)1,189 / 1,18993.4% / 90.7%99.4% / 97.8%98.2% / 97.2%0.879 / 0.8690.975 / 0.969L0 13.5 / 5.4 vs. 14.9 / 7.7; L1 30.9 / 21.9 vs. 32.8 / 27.5
Kimi-K3 judgeKimi-K3 (L0 / L1)1,507 / 1,50790.4% / 90.9%96.9% / 95.4%95.6% / 94.6%0.848 / 0.8700.964 / 0.959L0 24.4 / 17.6 vs. 27.9 / 23.7; L1 34.1 / 29.7 vs. 39.1 / 36.6
Kimi-K3 judgeQwen3.6-27B (L0 / L1)1,463 / 1,39589.7% / 85.9%98.2% / 96.6%97.0% / 94.4%0.845 / 0.8160.961 / 0.946L0 20.4 / 11.1 vs. 22.4 / 16.5; L1 28.1 / 16.9 vs. 31.9 / 23.4
  1. The knowledge-only property and the human baseline (Q2):
    • Matched K0 controls (already reported in §5 and Table 2): The 148 same-editor K0 edits carry no factual contradiction, and for 26 of 36 models they are flagged less often than authentic images are falsely rejected.
    • Human audits (Appendix C): Appendix C reports the construction-time audits (a blind spot-check of 99 forgeries, a full review of 194 absent-celebrity items, and 61 small-edit and near-duplicate cases). In addition, a human reviewer given the recorded fact verified 96.3% (52/54) of a random sample as having valid ground truth.
AuthorsOpenReview ↗

response part 3

2. Independence of Benchmark Construction/Grading and Human Baseline With vs. Without the Fact (Response to W3 & Q2) (continued)

  • Two-stage blind human baseline: On 60 items from 60 distinct canonical scenes (36 K1–K6 forgeries, 6 per family; 12 K0 controls; 12 untouched originals), a human evaluator judged each 1280 px image first blind (Stage 1) and then immediately after receiving a reference card with the canonical scene/date, the relevant historical or compositional fact, and a local zoom (Stage 2). If the forgeries could be caught without the specific fact, humans would detect them in Stage 1. Without the fact, humans detect 25.0% (9/36) of K1–K6 forgeries, mostly K2 cross-century artifacts that general historical common sense catches (5/6 = 83.3%); the event- and artwork-specific families (K1, K3–K6) average 13.3% (4/30). The reference card raises K1–K6 detection by +63.9 points to 88.9% (32/36, p=2.38×10−7p = 2.38 \times 10^{-7}) and balanced accuracy from 62.5% to 77.8% (+15.3 points). K0 controls, which go through the same Stage 2 protocol and zoom, stay at 0.0% (0/12) in both stages (Fisher's exact p=3.0×10−8p = 3.0 \times 10^{-8} vs. 88.9% on K1–K6):
Subset (60 distinct scenes)nnStage 1 (without fact)Stage 2 (with fact)ChangeExact two-sided McNemar pp
K1–K6 forgeries (Det)3625.0% (9/36)88.9% (32/36)+63.9 pp (+23 / −0)2.38×10−72.38 \times 10^{-7}
K1 People / K2 Objects / K3 Landmarks (Det)6 / 6 / 633.3 / 83.3 / 16.7%100.0 / 100.0 / 100.0%+66.7 / +16.7 / +83.3 pp—
K4 Text / K5 Attributes / K6 Canonical (Det)6 / 6 / 616.7 / 0.0 / 0.0%66.7 / 83.3 / 83.3%+50.0 / +83.3 / +83.3 pp—
K0 anchor-free controls (Det)120.0% (0/12)0.0% (0/12)0.0 pp (+0 / −0)1.00
Untouched originals (Auth)12100.0% (12/12)66.7% (8/12)−33.3 pp (+0 / −4)0.125 (n.s.)
Balanced accuracy (K1–K6 and authentic)4862.5%77.8%+15.3 pp—

3. Removing K0 Controls from the KGR Denominator (100% Ceiling, Response to W4)

Main-text Table 2 will use the 1,359 knowledge-anchored forgeries (K1–K6, excluding the 148 K0 controls) as the denominator for the primary KGR and E4 columns. This gives a 100.0% ceiling, with K0 reported separately. The human baseline is in §2 above.

Because K0 items carry no factual contradiction, excluding them scales every model's score by 1507/1359≈1.109×1507 / 1359 \approx 1.109\times (minus any rare K0 false grounding). Under the 100%-ceiling K1–K6 denominator, only one of the 36 models (Gemini-3.8-Flash) reaches 50% KGR (51.58% under L0, 60.41% under L1; E4 31.49% and 41.13%), and the second-best (GPT-5.6) reaches 39.81% KGR / 29.14% E4 under L1. Values in %; K1–K6: nn = 1,359; All: nn = 1,507 (90.2% ceiling); K0: nn = 148:

ModelL0 KGR (K1–K6)L0 E4 (K1–K6)L0 KGR (All)L1 KGR (K1–K6)L1 E4 (K1–K6)L1 KGR (All)L0 K0 false grounding
Gemini-3.8-Flash51.5831.4946.5860.4141.1355.140.68
GPT-5.621.939.2019.7739.8129.1435.970.00
GLM-5.3-Flash32.7524.9529.7338.2632.8234.772.03
Kimi-K327.0119.5024.4237.6032.7534.110.68
Kimi-K2.624.5816.7822.1635.3928.3332.050.00
Qwen3.5-122B-A10B26.0514.7923.5634.7324.1431.520.68
Qwen3-VL-235B-A22B20.0912.8818.1229.7318.7626.940.00
Qwen3.6-27B21.8511.9219.7728.7017.3726.150.68
Qwen3.5-27B19.7911.6317.9227.5219.4324.950.68
Qwen3.8-27B14.648.0213.3424.5015.7522.301.35
Qwen3.6-35B-A3B19.4310.7417.5923.9915.9721.770.68
Gemma-4-31B12.955.7411.6821.3411.9219.440.00
Qwen3.5-9B12.736.7711.4816.9210.1615.260.00
MiniMax-M36.623.835.9714.7210.6713.270.00
InternVL3.5-30B-A3B6.111.335.5111.632.8710.480.00

Comments on the paper

4 comments
AuthorsOpenReview ↗

Clarification: The Two p-Values Are Labeled "Det" (Not E4) and Reproduce Exactly from Released Logs

Because this note raises a serious concern about the integrity of our work before we had an opportunity to respond, we wish to clarify immediately that the two pp-values in question are explicitly labeled "Det" and test forgery detection, rather than E4.

Specifically:

  1. The two pp-values test "Det", not "E4": In Part 1 (Kimi-K3, Crop+Caption), the cell is explicitly labeled "Det 4.0×10−84.0 \times 10^{-8} / 9.1×10−139.1 \times 10^{-13}"—testing binary forgery detection (Det), where Kimi-K3 detects 52/145 (Full), 39/145 (Full+Caption), and 91/145 (Crop+Caption) forgeries. The calculation in the note applied those pp-values to the E4 counts (3, 2, and 10) from a different column. From the saved item-level outcomes, scipy.stats.binomtest(n01, n01 + n10, 0.5) reproduces both reported Det pp-values to every displayed digit:
Kimi-K3 Det comparison (n=145n = 145)Both fail (00)Gain (01)Loss (10)Both hit (11)Exact two-sided McNemar pp
Full (5252) →\to Crop+Caption (9191)47467454.0032×10−84.0032 \times 10^{-8}
Full+Caption (3939) →\to Crop+Caption (9191)50564359.0845×10−139.0845 \times 10^{-13}

(For E4 [3→103 \to 10 and 2→102 \to 10], the discordant counts are 9/2 [p=0.0654p = 0.0654] and 9/1 [p=0.0215p = 0.0215], which were never assigned the Det pp-values above.)

  1. Every other flagged percentage is likewise an exact integer ratio (full breakdown in our direct reply below):
    • Table 17 & GPT-5.6 Crop+Caption authentic accuracy: Full evaluates the 98 K1-scene originals, whereas the paper's Crop / Crop+Caption columns pool all 565 authentic crop boxes defined by peripheral edit windows across 191 authentic originals (Qwen3-VL-2B: 562/565=99.47%562/565 = 99.47\% [99.5%]; GPT-5.6: 474/565=83.89%474/565 = 83.89\% [83.9%] Auth, 76.77%76.77\% [76.8%] Bal). Our rebuttal's scale-matched table uses 1 matched K1 crop per scene across those 98 K1 originals (85/98=86.73%85/98 = 86.73\% Auth, 78.19%78.19\% Bal) to match Pad+Caption (n=98n = 98) 1-to-1, with K1 forgery Det identical in both (101/145=69.66%101/145 = 69.66\%).
    • Qwen3.6-27B FACE-pass rates: Of the 47/119 FACE-pass items, 43 produce a completed Full (L0) verdict (30/43=69.8%30/43 = 69.8\% REAL, 13/43=30.2%13/43 = 30.2\% naming the erased person), while 4 hit the 30,000-token cap inside <think> repetition loops without a verdict.
    • Peripheral sum & rounding: K1–K5 peripheral items sum to 781 (145+294+89+108+145145+294+89+108+145), plus 121 peripheral K0 controls = 902 total; Qwen2.5-Omni-7B and Qwen3.5-4B both have 177/1,507=11.7452%177/1{,}507 = 11.7452\% L1 KGR (11.75%→11.8%11.75\% \to 11.8\% in Table 5 vs. direct 11.7%11.7\% in Table 15).

All underlying item-level model outputs, paired contingency tables, and reproduction scripts are available for inspection at https://anonymous.4open.science/r/material-DCEB/. We are concerned that Reviewer s6Se told the full review panel that these two p-values were “mathematically impossible” before we had an opportunity to respond. The paired detection counts above reproduce both values. Because the reviewer cited this claim when raising the possibility of fabrication and lowering the rating to 1, we ask Reviewer s6Se to correct the record in the same thread and reconsider the rating in light of that correction.

Reviewer_s6SeOpenReview ↗

Thanks for the prompt response and clarification

Thank you for the detailed clarification. I rechecked the original rebuttal and acknowledge that the two disputed p-values were explicitly labeled “Det,” not E4. Applying the E4 counts to those tests was my mistake. The supplied detection contingency tables reproduce both reported p-values, and I withdraw my statement that they were mathematically impossible, along with the associated suggestion of fabrication.

I will delete the note so that the other reviewers , ACs and SACs are not influenced.

AuthorsOpenReview ↗

Rebuttal Summary (1/2): Shared Strengths and Per-Reviewer Discussion Outcomes

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair, Senior Area Chair, and all five reviewers for their thoughtful feedback and engagement. Across a large five-reviewer panel, all five reviewers recognize the value of Mandela-Bench's problem formulation and counterfactual evaluation design, four of the five now recommend acceptance (ratings: 4, 3, 3, 3)—including Reviewers Pt6e and s6Se, who both raised their ratings to 3 following our follow-up responses and numerical clarification—and three reviewers (4yrk, Pt6e, s6Se) rate Soundness at the maximum 3/3. Reviewer 7net (rating: 2) confirmed in Round 2 that our 100%-ceiling KGR denominator and release-ethics safeguards resolve both concerns, and our subsequent Round-2 controls address all four remaining questions (spotlighted in Part 2 below).

Shared Strengths Highlighted Across Reviews

  • Novel & Practical Problem Formulation: Targets seamless, knowledge-only manipulations of canonical images where pixel forensic traces and image–caption mismatches provide little reliable signal [4yrk, DTnC, Pt6e, s6Se, 7net].
  • Discovery of the MLLM "Mandela Effect": Documents a striking failure pattern across 36 MLLMs (0.8B to frontier scale) where recognizing a familiar scene biases verification toward memory over visual evidence—naming erased figures and accepting edited photographs even when replacement faces are rejected in isolation [Pt6e, 4yrk, s6Se, DTnC, 7net].
  • Rigorous Counterfactual Controls: Pairing K1–K6 forgeries with same-editor K0 controls and untouched originals "neutralizes the two obvious shortcuts: editor artifacts and 'famous means real'" [7net, 4yrk, DTnC, s6Se].
  • Multi-Probe Diagnostic & Calibrated Scoring: Combining seven probes (L0, L1, L2, L-adv, RECOG, TEXT, FACE) with the human-calibrated E0–E4 ladder and KGR separates knowledge, perception, recognition, and verification [4yrk, DTnC, Pt6e, s6Se, 7net].

1. Per-Reviewer Summary: Positive Assessments (4yrk, Pt6e, s6Se, DTnC)

Reviewer & ScoreCore Concerns RaisedCompleted Rebuttal Evidence & Discussion Outcome
4yrk — 4 (Clear accept; Soundness 3/3, Info 3/3)Canonical single-fact scope (W1); subtle editing cues (W2); crop confounds (W3); KGR vs. E4 & framing (W4); mitigation (W5); subset CIs & effect sizes (W6).Clarified retrieval-free canonical scope; 50–69 pp same-seed swap-over-K0 gaps + detector regressions (p<0.05p < 0.05, 16/17 models); Pad+Caption + 13,660-row Full regression (OR=0.65,p=0.0004\text{OR} = 0.65, p = 0.0004); reported E4 alongside KGR; training-free Self-Captioned Crop (+13.8 to +26.9 pp Det, p≤0.0066p \le 0.0066, higher Bal & E4, no GT captions); 10k-bootstrap 95% CIs and Cohen's hh.
Pt6e — 2 →\to 3 (Weak accept; Soundness 3/3, Info 3/3)Gemini-3.8-Flash blur sweeps (W1, Q1); <think> audits on E2 and FACE-pass + rare framing for 1948 Truman (W2, Q2); prose fluency (W3).On Gemini-3.8-Flash, Target-Blur drops KGR 16/60 (26.7%) →\to 1/60 (1.7%) by σ=8–16\sigma = 8\text{--}16 px while Context-Blur keeps 30.0%–36.7% KGR (retaining all 18/18 baseline items at σ=2,8,16\sigma = 2, 8, 16 px and Pad); Cohort 2 shows a 20%–28% erased-person naming plateau at σ∈{4,8,16}\sigma \in \{4, 8, 16\} px; 97.7% of 2,114 E2 <think> logs localize discrepancies; Qwen3.6-27B's 43 FACE-pass <think> logs show 6/30 canonical reinstatement, 23/30 alternative-identity/mismatch features, and 0/30 rationalizations (Truman [1.6%] framed as rare; corpora to be released). Confirmed "no further concerns" and raised rating from 2 to 3.
s6Se — Raised to 3 (Weak accept; Soundness 3/3, Pres 3/3)Crop confounds (W1); editor & gate generalization (W2); K2–K5 and central (C1) de-recognition (W3); E4 (W4); paired counts & denominators.Cross-editor check on 96 paired Qwen-Image-2.1 edits + reverse gate check on 24 new Qwen-only K1 forgeries (unchanged ranking); 8-model K1–K5 Crop+Caption (781 K1–K5 + 121 K0 = 902 peripheral items) + 140-item central-figure (C1) Pad/Crop+Caption study (E4 up 2.4×–4.5×2.4\times\text{--}4.5\times); released item-level logs and contingency tables at material-DCEB. Confirmed calculations and raised rating to 3.
DTnC — 3 (Weak accept; Soundness 2/3)Event priors vs. visual scene context (W1, Q1); attack-aware defender (W3, Q2); E4 across families + 1:1 balance (W2, W3, Q3).Masking visual context (Pad+Caption vs. Full+Caption) raises KGR & E4 under identical event text and target scale; full-image Attack-Aware Prompt gives no consistent KGR gain (88.8% false alarms on Qwen3.6-27B) whereas Self-Captioned Crop raises Det, Bal, and E4 across all tested models; 368:368 1:1 subset preserves tiering (Gemini 1st: 87.4% Acc / 87.3 F1).
AuthorsOpenReview ↗

Rebuttal Summary (2/2): Spotlight on Completed Round-2 Controls for Reviewer 7net

2. Spotlight — Reviewer 7net (Rating: 2, Soundness: 1/3): Resolved Items & Completed Round-2 Controls

In Round 2 (18 Nov, 03:02), Reviewer 7net explicitly confirmed that our 100%-ceiling K1–K6 denominator (n=1,359n = 1{,}359, W4) and our Ethics Statement correction + three release safeguards (gated DUA access, C2PA provenance manifests, invisible forensic watermarking) resolve both concerns, while noting four remaining methodological questions. Later that same day (18 Nov, 13:58–14:01, Parts 1–3), we provided direct experimental controls addressing each of the four remaining points:

  1. Why Kimi-K3 Pad (33.10% Det) differs from Crop (62.76% Det), and full-subset / 7-model coverage: Clarified that the 145 items are 100% of the released peripheral K1 subset (not a sample), alongside 98 authentic controls. Kimi-K3's extra Crop+Caption FAKE verdicts are ungrounded (31 →\to 79 E1+E2), whereas fixed-1280 px Pad+Caption yields more grounded detections (KGR 17 vs. 12; E4 13 vs. 10, up from 3 and 2 under Full+Caption). Expanding Pad+Caption to 7 models shows that 5 of 6 nonzero-baseline models raise both KGR and E4 at fixed 1280 px scale and 6 of 7 raise Bal (the planned §5 revision separates matched-scale context-masking gains from crop-induced ungrounded FAKE shifts).
  2. Scale-matched FACE control across all 119 pairs: Holding the 1280 px canvas, unrotated native face pixels, red target box, and identity prompt fixed (Pad-FACE vs. Full-FACE), Pad-FACE differs from Crop-FACE by 2–3 pairs (85 vs. 88 on Gemini, 42 vs. 44 on Qwen3.6-27B, 21 vs. 24 on refusal-audited Kimi-K3.1-Pro; p=0.678–0.868p = 0.678\text{--}0.868), yet models still judge the Full forgery REAL on 28.2% (24/85, Gemini) and 76.2% (32/42, Qwen) of Pad-FACE-passing items under L0 (11.8% and 69.0% under L1)—confirming the cross-task verification gap without crop magnification.
  3. Construction- and roster-independent explanation judge (DeepSeek-V4.1-Flash): Beyond calibrated judge–candidate overlap practices in recent VLM benchmarks (SpatiaLQA, CVPR 2026; Text2Vis, EMNLP 2025), we ran DeepSeek-V4.1-Flash—uninvolved in benchmark construction and outside the original 36-model roster (and excluded from grading its own Pad outputs)—finding 93.2% (192/206) KGR agreement with blind human labels (E0–E4 κw=0.897\kappa_w = 0.897), 96.8% raw KGR agreement with Qwen3.8-27B across 400 stratified archived answers, and unchanged L0/L1 point-estimate rankings.
  4. Evaluator disclosure & 2×22 \times 2 (Fact ×\times Zoom) blind human study (8 raters, 120 new scenes, 960 trials): Disclosing that the Round-1 60-scene pilot used one label-blinded co-author aware of the hypothesis and bundled the card with zoom, we ran a counterbalanced 2×22 \times 2 study across 120 disjoint scenes with 8 hypothesis-blind external graduate-student evaluators: Fact-only (no zoom, no box) raises K1–K6 detection from 25.0% to 84.7% (+58.3 pp over Second look 26.4%, Bal = 90.3%), whereas Zoom-only reaches 27.8% (+1.4 pp), and same-editor K0 controls remain at 0/48 (0.0%) in all four arms.

Closing

We are deeply grateful to Reviewers 4yrk, Pt6e, s6Se, and DTnC for their positive assessments of Mandela-Bench, and to Reviewer 7net for the rigorous methodological feedback that motivated the matched-scale, external-judge, and 2×22 \times 2 blind human controls above. All new experiments, E4 and 100%-ceiling tables, denominator clarifications, release safeguards, and prose improvements will be incorporated into the final manuscript. Thank you once again to the Area Chair, Senior Area Chair, and all reviewers for your time and dedicated service.

Best regards, The Authors of Submission 11998

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.