- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces a new benchmark DECOY to investigate whether existing MLLMs can answer questions about less salient evidence in the presence of salient completing objects. The benchmark contains 2163 images from four eye-tracking datasets and 9596 human-written question-answer pairs across five task types. The evaluation measures Saliency Grounding (SG), Target Grounding (TG), and Answer Accuray (AA). A suppression failure is also defined as an incorrect answer conditional on correct SG and TG. The authors report performance across 24 models which shows that substantial conditional answering errors and a positive relationship between fixation-derived saliency gaps and model errors. Distractor attenuation reduces errors, distractor removal, target crops, and region-scoped question rewriting recover considerably more failures than location hints. Finally, probing and activation patching suggest that answer-relevant information remains decodable while the effectiveness of intervening on visual-region states decreases with decoder depth.
Critical strengths
- The paper explores an important direction of finding evidence and also answer from it. Existing MLLMs report whether the final answer is correct without examining what intermediate capabilities are present. DECOY separately evaluates saliency identification, evidence localization in addition to just answering.
- The paper uses human fixation during data collection, which provides a model-independent measure of visual competition. The saliency gap is derived from eye-tracking data rather than defined using the evaluated models’ errors or attention maps. This reduces the ambiguity of whether the distractor is strong or the model simply fail in tis presence.
- The benchmark construction is substantial and includes useful annotation checks. The dataset contains 2,163 images and 9,596 human-written questions across five task types, with multiple annotators and adjudication of disagreements.
- The within-item attenuation experiment provides stronger evidence that changing the competing region affects model behavior. The authors progressively attenuate the distractor while keeping the question, target, answer, and candidate set fixed. They also include a matched irrelevant-region editing control and human manipulation checks. The reduction in mean error from 61.4% to 47.9%, compared with 61.2% under the neutral edit, supports the conclusion that changing the competing region affects model behavior.
Critical weaknesses
- The paper shows that a model can identify the relevant location in one query and answer incorrectly in another, but this does not prove that it found and read the relevant evidence during the query where it answered incorrectly. SG, TG, and AA are evaluated through independent queries with different instructions and candidate sets. Correct TG therefore demonstrates that a model can select an appropriate location when explicitly asked to do so. It does not establish that the same localization occurred during the AA computation.
- Preservation of the original task under removal and rewriting is not sufficiently demonstrated. Take Figure 1 for example, The correct answer of two numbers greater than five appears to require both “79” on the distractor sign and “446” on the train. The sign therefore contributes necessary evidence to the original question. Removing it or restricting the question to the train region, would change the correct count.
- The experiment of content probe does not identify the contribution of target-region representation. The probe combines pooled target-region states with contextualized candidate representations extracted from the AA pass, which may already contain information from the image and question. Although the reported 70.5% accuracy establishes decodability from a joint representation, but does not show how much useful information is provided by the target-region component.
- The paper is not self-contained, with some tables in the appendix but the corresponding text description in the main paper.
- The experiments only partially separate perceptual saliency from semantic competition. Distractor attenuation changes chroma, contrast, and spatial detail together. These edits may reduce visual prominence, but they may also weaken the recognizability of the distractor-linked answer. For example, changing the color of a competing object can directly alter the attribute that makes an incorrect color option attractive.
Questions for authors
- Regarding the example in Figure 1, is the example included in the intervention experiments? How to verify that removal and region-scoped rewriting preserve the correct answer, particularly for counting and compositional questions?
- It would be better to supplement the independent-query protocol with an evaluation that incorporate localization and answering within the same inference sequence? How would its results change the interpretation of SFR?
- Regarding the probe attribution, what performance is obtained from candidate representations alone? What happens when target-region features are replaced with shuffled features or matched non-target features?
4 comments
Responses to Reviewer GcQB (Part II)
-
W3 & Q3: The requested ablations are added to Table 2(b) and Table R10 (Appendix K.10), with the same folds as the original probe. Accuracy averaged over the seven models is 70.5% for the original probe. It falls to 46.9% with candidate representations alone, 46.6% with target states shuffled across questions, and 46.8% with a matched non-target region. The target component therefore supplies the information that lifts the probe from 46.9% to 70.5%, and this information is specific to the target region.
On the concern about contextualized candidates, Text cand. pairs the target states with candidate representations computed without the image. It still reaches 62.9%. Most of the accuracy is therefore provided by the target region and not by image information inside the candidate representations. These results are added to Section 5.2.2.
-
W4: We agree that several results were previously reported in the main text only through pointers to the appendix. The revision addresses this in three ways.
(1) The new results relevant to the main claims now appear in main text tables. Table 1 adds the DL column and a panel with the mean SG, TG, and AA of each task type. Table 2(b) adds two probe ablations, Table 3 adds the Rewrite only control, and the new Table 4(c) reports localization and answering in one sequence.
(2) Every result in Appendix K is stated in the main text with its key number and a pointer to the full table.
(3) The main text now uses the full ten pages.
-
W5: Attenuation is repeated with the chroma scale fixed at 1.0, so the chroma of the distractor is kept and only contrast and blur vary (Table R12, Appendix K.12). The same 318 images and 1,297 questions are used.
Answer error, L0 to L3 (%) Human distractor preference, L0 to L3 (%) Original operator 61.4 to 47.9 73.6 to 48.1 Chroma fixed 61.4 to 53.1 74.0 to 57.3 Error still falls by 8.3 points with the chroma of the distractor unchanged, so the effect does not depend on altering the attribute that makes a color option attractive. The preference ratings come from a separate panel of 24 raters. Answer error and human preference fall in about the same proportion under both operators. Error therefore follows how strongly the distractor draws attention. This result is added to Section 4.3.
We hope these results address the reviewer's concerns, and we are happy to discuss any further questions.
Response for Authors
Thanks for the responses. My follow up question for W1 is that the results can show that the model has correct localization but wrong answer prediction, it does not tell whether it is because the model fail to extract the relevant information or fail to use it? For W5, although the chroma of the distractor is unchanged, blur and lower contrast could also make the distractor less attention-catching or harder to recognize. The experiment does not separate those two effects.
Responses to the Follow-up Questions
To reviewer GcQB: We appreciate the follow-up questions. Responses are given below.
-
W1 (extracting or using the evidence): The internal analyses separate the two. They show that the evidence is extracted into the target-region representation, and that the failure arises when the answer is formed.
(1) The target-region states carry the answer even when the model answers wrong. The content probe reads the correct answer from the target-region states on 70.5% of the questions the model answers correctly, and also on 70.5% of those it answers wrong (Table 4(b)). This information comes from the target region itself. Without the target states, or with shuffled or matched non-target states, probe accuracy falls to about 47% (Table R10).
(2) Replacing the target-region states does not repair the answer. Patching uses the model's own Removal run, which answers correctly, as the source. Replacing the target-region states of the failed run with those of this correct run recovers only 25.3% of the failures at the shallowest depth. Replacing the distractor-region states recovers 72.6% (Figure 5(a), Table R11). The target-region states of a failed run therefore already carry most of what the correct run needs, and the failure is driven by the distractor region.
(3) The readout and the answer fail differently. When the probe errs, 34.0% of its errors select the distractor-linked option, close to the 33.3% expected by chance. The same models choose that option in 48.6% of their own errors (Section 5.2.2). The pull toward the distractor appears in the answer, and not in the extracted evidence.
The evidence is extracted, and the model fails to use it, as stated in Section 5.2.2.
-
W5 (attention or recognizability): To separate the two effects, we ran an additional recognition test on the same 1,297 questions from 318 images. At every attenuation level, each of the seven models identifies the distractor object in a four-way question, with chance at 25%. Neutral applies the same edit to an irrelevant region.
(a) Distractor recognition accuracy (%)
Model L0 L1 L2 L3 Neutral mplug-owl-7B 94.1 93.1 92.8 91.8 94.4 Qwen3-VL-2B 94.4 93.7 92.2 92.1 94.6 LLaVA-v1.5-7B 94.4 94.3 93.6 93.1 94.4 Qwen3-VL-8B 95.7 94.6 93.8 93.8 96.3 Qwen3-VL-8B-Thinking 95.8 96.0 95.1 94.4 95.5 InternVL3.5-8B 95.8 95.4 94.3 92.9 95.6 MolmoPoint-8B 96.7 96.0 95.4 95.3 96.7 Macro mean 95.3 94.7 93.9 93.3 95.4 (b) Answer error (%) on the questions whose distractor each model still recognizes at L3
Model Questions L0 L3 Reduction mplug-owl-7B 1,190 68.2 54.3 13.9 Qwen3-VL-2B 1,194 66.2 52.0 14.2 LLaVA-v1.5-7B 1,207 64.0 50.7 13.3 Qwen3-VL-8B 1,216 60.4 47.0 13.4 Qwen3-VL-8B-Thinking 1,224 58.8 45.9 12.9 InternVL3.5-8B 1,205 55.9 43.3 12.6 MolmoPoint-8B 1,236 52.6 40.7 11.9 Macro mean 1,210 60.9 47.7 13.2 All questions (Table 19) 1,297 61.4 47.9 13.5 Recognition falls by only 2.0 points from L0 to L3, while answer error falls by 13.5 points. The distractor remains recognizable at every level, and the edit on an irrelevant region leaves recognition unchanged. On the questions whose distractor is still recognized at L3, error falls by 13.2 points, close to the full effect of 13.5 points, and every model shows a reduction of at least 11.9 points. Reduced recognizability therefore does not explain the attenuation effect, which follows the drop in attention.
We hope these responses address the follow-up questions, and we are happy to discuss further.
Responses to Reviewer GcQB (Part I)
To reviewer GcQB: We appreciate the careful reading and the constructive questions. Responses to specific comments are given below, and the corresponding revisions are highlighted in blue in the revised paper.
W1 & Q2: Three results now place the localization inside the answering computation.
(1) Localization and answering in one sequence. Ten models select a TG option and then an AA option in one generation (Table 4(c) in the main text, Table R7 in Appendix K.7). TG in this protocol reaches 81.1% on average.
SFR stays close to its original value, and the ranking of the ten models is unchanged. The failure persists right after the model has stated the correct location in its own output.
(2) Localization in the AA forward pass. The where probe is refitted on the forward pass that produces the AA answer. It identifies the evidence region in 79.5% of the wrong answers and 79.8% of the correct ones (Table R7(b)). The location is encoded during the AA computation whether or not the answer is correct.
(3) The model's own localization as input. Table 3 of the original submission already returns the model's own correct TG answer together with the AA question (TG fb.). This condition recovers 3.8% of the failures. Supplying the localization in the same context does not repair the answer.
The independent protocol prevents SG and TG from acting as hints for AA, and result (1) shows that this choice changes SFR only slightly. Results (1) and (2) are added to Sections 4.2.1 and 5.2.2.
W2 & Q1: We agree with the reviewer's reading of the original Figure 1. Its count draws on both the sign and the train, so the sign also carries evidence. Figure 1 is replaced with an item from the failure set of Claude-Opus-4.8. The question asks which metal vessel rests on the cork trivet. The model identifies the black cat as the most salient object and selects the region containing the kettle, yet answers with the frying pan under the cat. Removal and region rewriting both return the kettle, and a location hint does not (Figure 1(a)).
To verify answer preservation at scale, a stratified sample of 461 items is audited (Table R9, Appendix K.9). Two annotators judge independently whether the gold answer still holds after Removal and after region rewriting, and a third resolves disagreements. Cohen's κ is 0.83 and 0.86.
Counting and Reasoning questions are the most affected, as the reviewer expected. An invalid item is still scored against its original gold answer, so a correct reading of the edited input counts as unrecovered. Excluding the invalid items therefore raises recovery, and Removal + Crop rises from 69.2% to 71.3% in the same way. The recovery reported in Table 3 is thus a conservative estimate. The audit is added to Section 5.1.2.