- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PlotTwist is a 75-item visual riddle dataset (38 collected online and 37 generated from human-designed scenarios) built to test whether vision-language models can weigh competing pieces of visual evidence within a single scene. Each image pairs a locally plausible misleading clue that supports a tempting wrong answer with stronger diagnostic evidence that supports the intended answer, both localized with bounding boxes. The authors evaluate 14 models on the unmarked images (41.9% average accuracy) and then re-query them after highlighting the misleading clue when the initial answer was right, or the diagnostic clue when the initial answer was wrong. While diagnostic highlights help models recover from mistakes, misleading highlights frequently cause them to flip initially correct predictions.
Critical strengths
-
Separating statistical background shortcuts from within-image evidence arbitration is a well-motivated angle. In many visual puzzles, a distracting detail is not noise; it is a semantically relevant observation that simply gets outweighed by stronger evidence elsewhere in the scene.
-
Evaluating correction alongside regression gives a much more honest view of visual prompting than reporting net accuracy alone, showing that apparent gains from visual hints can coexist with high vulnerability to misleading salience.
Critical weaknesses
-
The PDF appears to contain a few unfinalized draft notes that looks like LLM edits. It should be easy to clean up in the revision (and several figures have tiny, hard-to-read text). Sections 5.2 and 5.3 refer to "the supplied draft" and note that the baseline numbers in Table 1 do not match Table 2 ("pending reconciliation"), while Appendix A.2 notes that "the supplied manuscript" omits temperature and whether the second pass included the first-pass conversation history. Knowing if Pass 2 is single-turn or multi-turn is important for interpreting persistence and regression, and one of the 75 items is currently unscored (with Figure 5 also listing a few zero-box items). These details all look straightforward for the authors to clarify during the rebuttal.
-
The intervention design leaves a few confounds open. Because models only get the misleading hint when right at baseline and the diagnostic hint when wrong, each model's transition rates are measured on a different subset of items. Since 74 items cost only a few dollars to run, testing all models on both hints across every item (and separating bounding boxes from the edge-panel zoom) would make the comparisons much cleaner. It would also help to check whether wrong answers actually match the specific choice tied to the misleading clue.
Questions for authors
-
Was an intermediate draft or automated edit uploaded by mistake? If you can share the missing Pass 2 setup details (especially whether it was single-turn or multi-turn and how zero-box items in Figure 5 were handled) and reconcile Table 1 with Table 2, that would resolve my main concern about the submission.
-
When models get an item wrong at baseline or regress after a misleading hint, how often do they choose the specific incorrect option linked to the annotated misleading clue versus another distractor?