acceptodds
Under review as a conference paper at ICLR 2027

PlotTwist: Deception-Aware Reasoning in Visual Riddles

Abstract

Large vision-language models (LVLMs) have achieved remarkable progress in visual understanding, yet their ability to reason reliably in the presence of misleading visual information remains largely unexplored. Existing benchmarks mainly evaluate recognition and question answering in settings where relevant visual evidence is explicit and does not intentionally conflict with distracting cues. However, real-world visual interpretation can involve misleading signals and deceptive patterns that provide compelling but unreliable evidence, making it important to understand whether multimodal models can identify trustworthy evidence and resist deceptive interpretations. Unlike conventional spurious-correlation settings, where models exploit unreliable statistical associations, the misleading clues considered here are semantically relevant and locally plausible. The challenge is therefore to arbitrate among competing pieces of evidence and prioritize the most diagnostic visual cues. We introduce PlotTwist, a diagnostic benchmark for evaluating deception-aware reasoning in visual riddles. PlotTwist focuses on scenarios in which models must distinguish diagnostic evidence from deceptive clues, prioritize informative visual cues, and infer the intended answer beyond superficially plausible interpretations. To obtain sufficiently diverse and creative examples, PlotTwist combines visual riddles curated from online sources with newly developed riddles created through human-designed scenarios with the aid of generative models, which also support image generation. This process enables the construction of diverse visual riddles exhibiting different forms of misleading evidence and deception. PlotTwist contains 75 visual riddles, accompanied by structured annotations of deceptive visual clues, diagnostic evidence, intended answers, and solution rationales. We evaluate state-of-the-art proprietary and open-weight LVLMs by first measuring their ability to solve the riddles. We then use manually annotated regions to construct visual hints that selectively emphasize either deceptive clues or diagnostic evidence. By comparing model responses before and after these interventions, we examine how strongly models persist in their initial interpretations, whether emphasizing deceptive evidence can induce incorrect answers, and whether highlighting diagnostic evidence can recover initially incorrect predictions. Our experiments show that current LVLMs are frequently misled by plausible deceptive cues and often fail to identify or sufficiently prioritize the evidence required for the intended solution. PlotTwist provides a systematic framework for studying how vision-language models reason under misleading visual evidence and offers a challenging benchmark for developing more robust, evidence-grounded, and deception-resistant multimodal systems. The benchmark and code are available at https://asdafi.github.io/PlotTwist.

Primary area:
datasets and benchmarks
Keywords:
Large Vision-Language Models, Multimodal Reasoning, Multimodal Benchmarks, Deception-Aware Reasoning, Misleading Visual Cues, Visual Riddles

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_SdrmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PlotTwist is a 75-item visual riddle dataset (38 collected online and 37 generated from human-designed scenarios) built to test whether vision-language models can weigh competing pieces of visual evidence within a single scene. Each image pairs a locally plausible misleading clue that supports a tempting wrong answer with stronger diagnostic evidence that supports the intended answer, both localized with bounding boxes. The authors evaluate 14 models on the unmarked images (41.9% average accuracy) and then re-query them after highlighting the misleading clue when the initial answer was right, or the diagnostic clue when the initial answer was wrong. While diagnostic highlights help models recover from mistakes, misleading highlights frequently cause them to flip initially correct predictions.

Critical strengths
  1. Separating statistical background shortcuts from within-image evidence arbitration is a well-motivated angle. In many visual puzzles, a distracting detail is not noise; it is a semantically relevant observation that simply gets outweighed by stronger evidence elsewhere in the scene.

  2. Evaluating correction alongside regression gives a much more honest view of visual prompting than reporting net accuracy alone, showing that apparent gains from visual hints can coexist with high vulnerability to misleading salience.

Critical weaknesses
  1. The PDF appears to contain a few unfinalized draft notes that looks like LLM edits. It should be easy to clean up in the revision (and several figures have tiny, hard-to-read text). Sections 5.2 and 5.3 refer to "the supplied draft" and note that the baseline numbers in Table 1 do not match Table 2 ("pending reconciliation"), while Appendix A.2 notes that "the supplied manuscript" omits temperature and whether the second pass included the first-pass conversation history. Knowing if Pass 2 is single-turn or multi-turn is important for interpreting persistence and regression, and one of the 75 items is currently unscored (with Figure 5 also listing a few zero-box items). These details all look straightforward for the authors to clarify during the rebuttal.

  2. The intervention design leaves a few confounds open. Because models only get the misleading hint when right at baseline and the diagnostic hint when wrong, each model's transition rates are measured on a different subset of items. Since 74 items cost only a few dollars to run, testing all models on both hints across every item (and separating bounding boxes from the edge-panel zoom) would make the comparisons much cleaner. It would also help to check whether wrong answers actually match the specific choice tied to the misleading clue.

Questions for authors
  1. Was an intermediate draft or automated edit uploaded by mistake? If you can share the missing Pass 2 setup details (especially whether it was single-turn or multi-turn and how zero-box items in Figure 5 were handled) and reconcile Table 1 with Table 2, that would resolve my main concern about the submission.

  2. When models get an item wrong at baseline or regress after a misleading hint, how often do they choose the specific incorrect option linked to the annotated misleading clue versus another distractor?

Reviewer_kKnmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PlotTwist, a benchmark designed to evaluate deception-aware reasoning in large vision-language models (LVLMs). The benchmark contains 75 visual riddles collected from online sources and human-designed scenarios, where misleading clues compete with diagnostic evidence supporting the correct answer. The authors annotate these clues and their corresponding visual regions, and evaluate 14 LVLMs using both standard accuracy and targeted visual-hint interventions. The experimental results suggest that current LVLMs struggle with competing visual evidence and that highlighting different clues can significantly affect their predictions.

Critical strengths
  1. This paper investigates an important problem in multimodal reasoning: whether LVLMs can distinguish misleading clues from more reliable visual evidence.
  2. This paper investigates an important problem in multimodal reasoning: whether LVLMs can distinguish misleading clues from more reliable visual evidence.
Critical weaknesses
  1. This paper defines visual clues supporting the reference answer as Diagnostic Evidence and clues supporting other answers as Misleading Clues. However, the visual evidence in some riddles may not be sufficient to uniquely determine the correct answer. For example, in the killer identification task in Figure 1, hiding a knife does not necessarily mean that the person is the killer, and having blood on one's clothes is not necessarily misleading evidence. Therefore, the current evaluation may conflate the model's failure to properly weigh visual evidence with its failure to follow the intended solution of the riddle. Although this paper uses human review to ensure the plausibility of the reference answers, further validation of the relative evidential strength of different clues and the existence of other reasonable interpretations is still needed.

  2. The benchmark contains only 75 visual riddles, which seems too small to support reliable conclusions across 14 LVLMs. Moreover, the quality of the dataset remains unclear, as the paper lacks sufficient details about human annotation, inter-annotator agreement, and the validation of ground-truth answers. These limitations raise concerns about the reliability and generalizability of the benchmark.

  3. The experiments do not convincingly demonstrate that models make incorrect predictions because they are misled by deceptive clues. In particular, the visual-hint experiments lack appropriate control groups to isolate the effect of deception. When a model changes its answer after a misleading clue is highlighted, it is unclear whether this change is caused by the deceptive nature of the clue or simply by the increased visual salience of the highlighted region. Additional controlled experiments are needed to support the central claim.

  4. The reported baseline accuracies in Sections 5.1 and 5.2 are inconsistent, and the authors acknowledge that the underlying evaluation cohorts have not been reconciled. Furthermore, Section 5.3 contains statements such as "retained from the supplied draft" and "require verification," which appear to be unresolved editorial notes and suggest that some model-level results have not been properly verified before submission. These issues raise concerns about the reliability of the reported performance changes and weaken the credibility of the experimental conclusions.

Questions for authors
  1. How can it be objectively determined whether a visual clue is Diagnostic Evidence or a Misleading Clue? For example, in Figure 1, why does hiding a knife provide stronger evidence that someone is the killer than having blood on one's clothes? When multiple clues support plausible but uncertain interpretations, how can the evaluation ensure that a model choosing a non-reference answer represents a Deception-Aware Reasoning Failure rather than a reasonable alternative inference?

  2. Can the authors expand the benchmark and provide inter-annotator agreement results to better demonstrate its quality and reliability?

  3. Can the authors include additional control experiments to distinguish the effects of deceptive clues from those of increased visual salience?

  4. Can the authors explain the inconsistencies in the reported experimental results and clarify whether all model-level results have been properly verified?

Reviewer_U3z4OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces PlotTwist, a benchmark of 75 visual riddles designed to evaluate the ability of large vision-language models (LVLMs) to arbitrate between misleading clues and diagnostic evidence. Through targeted visual-hint interventions on 14 LVLMs, the authors find that current models are susceptible to misleading evidence and may struggle to consistently reach correct conclusions even when diagnostic evidence is highlighted.

Critical strengths
  1. The paper investigates evidence arbitration among competing yet semantically plausible visual clues, distinguishing its focus from traditional visual illusions and spurious correlations. This provides a meaningful perspective for understanding reasoning failures in LVLMs.

  2. The benchmark provides structured annotations of misleading clues, diagnostic evidence, and their corresponding visual regions, enabling more fine-grained analysis of model behavior beyond conventional accuracy metrics.

  3. The evaluation covers 14 LVLMs across different model families and scales. By examining both correction and regression under visual hints, the paper offers useful insights into model behavior beyond aggregate performance comparisons.

Critical weaknesses
  1. The novelty primarily lies in benchmark construction and the combination of existing visual prompting techniques. Given related work such as Visual Riddles, Sherlock, and VI-Probe, the paper does not sufficiently establish whether PlotTwist reveals reasoning failures that existing benchmarks cannot capture.

  2. The benchmark contains only 75 riddles, limiting the generalizability of the findings. Moreover, the lack of systematic human evaluation and inter-annotator agreement analysis raises concerns about the reliability of distinguishing misleading clues from diagnostic evidence, particularly given the potential ambiguity of visual riddles.

  3. The visual-hint experiments lack important controls, such as neutral highlighting, making it difficult to disentangle the effects of misleading evidence from general visual salience. Additionally, inconsistencies between baseline and intervention results, together with the absence of repeated trials and statistical significance analysis, weaken the reliability of the conclusions.

Questions for authors
  1. How do the authors validate that the annotated misleading clues are genuinely deceptive and that the diagnostic evidence supports a uniquely defensible answer? Could independent human evaluations and inter-annotator agreement analyses help establish the benchmark's validity?

  2. Could the authors introduce additional controls, such as neutral, misleading, and diagnostic highlighting, to distinguish general sensitivity to visual prompts from genuine failures in evidence arbitration? Could they also further disentangle perceptual failures from reasoning failures?

  3. Could the authors reconcile the evaluation samples used in the baseline and visual-hint experiments and report repeated trials, confidence intervals, or statistical significance tests to establish the robustness of the observed model differences and answer transitions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.