acceptodds
Under review as a conference paper at ICLR 2027

Knowing Where Without Reading What: Saliency-Induced Access Failure in MLLMs

Abstract

Visual question answering (VQA) requires models not only to locate task-relevant evidence but also to use it despite more salient competitors. Although multimodal large language models (MLLMs) excel at general VQA, their ability to exploit correctly localized evidence against salient distractors remains largely unexplored. To address this gap, we present **DECOY**, a benchmark designed to systematically evaluate MLLMs under conflicts between visual saliency and task relevance. DECOY contains 2,163 images from four eye-tracking datasets and 9,596 human-written question-answer pairs across five task types, each graded by a fixation-based saliency gap. Separate evaluations of *Saliency Grounding*, *Target Grounding*, and *Answer Accuracy* isolate suppression failures after correct grounding. Experiments on 24 MLLMs reveal that models know what is salient and where the evidence is, yet let the salient distractor dominate the answer: the lowest suppression failure rate is 37.7%, versus 4.9% for humans, and answering errors increase monotonically with the saliency gap. Beyond benchmarking, we uncover a gap between evidence localization and utilization. Restricting the input to target evidence is substantially more effective than indicating its location, and answer-relevant information remains decodable even as the causal influence of visual-region states fades with depth. These findings establish DECOY as a testbed for improving evidence utilization. The benchmark and code will be released upon publication.

Primary area:
datasets and benchmarks
Keywords:
Multimodal large language models, Visual saliency, Benchmark evaluation, Evidence localization, Mechanistic interpretability, Activation patching

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GcQBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a new benchmark DECOY to investigate whether existing MLLMs can answer questions about less salient evidence in the presence of salient completing objects. The benchmark contains 2163 images from four eye-tracking datasets and 9596 human-written question-answer pairs across five task types. The evaluation measures Saliency Grounding (SG), Target Grounding (TG), and Answer Accuray (AA). A suppression failure is also defined as an incorrect answer conditional on correct SG and TG. The authors report performance across 24 models which shows that substantial conditional answering errors and a positive relationship between fixation-derived saliency gaps and model errors. Distractor attenuation reduces errors, distractor removal, target crops, and region-scoped question rewriting recover considerably more failures than location hints. Finally, probing and activation patching suggest that answer-relevant information remains decodable while the effectiveness of intervening on visual-region states decreases with decoder depth.

Critical strengths
  1. The paper explores an important direction of finding evidence and also answer from it. Existing MLLMs report whether the final answer is correct without examining what intermediate capabilities are present. DECOY separately evaluates saliency identification, evidence localization in addition to just answering.
  2. The paper uses human fixation during data collection, which provides a model-independent measure of visual competition. The saliency gap is derived from eye-tracking data rather than defined using the evaluated models’ errors or attention maps. This reduces the ambiguity of whether the distractor is strong or the model simply fail in tis presence.
  3. The benchmark construction is substantial and includes useful annotation checks. The dataset contains 2,163 images and 9,596 human-written questions across five task types, with multiple annotators and adjudication of disagreements.
  4. The within-item attenuation experiment provides stronger evidence that changing the competing region affects model behavior. The authors progressively attenuate the distractor while keeping the question, target, answer, and candidate set fixed. They also include a matched irrelevant-region editing control and human manipulation checks. The reduction in mean error from 61.4% to 47.9%, compared with 61.2% under the neutral edit, supports the conclusion that changing the competing region affects model behavior.
Critical weaknesses
  1. The paper shows that a model can identify the relevant location in one query and answer incorrectly in another, but this does not prove that it found and read the relevant evidence during the query where it answered incorrectly. SG, TG, and AA are evaluated through independent queries with different instructions and candidate sets. Correct TG therefore demonstrates that a model can select an appropriate location when explicitly asked to do so. It does not establish that the same localization occurred during the AA computation.
  2. Preservation of the original task under removal and rewriting is not sufficiently demonstrated. Take Figure 1 for example, The correct answer of two numbers greater than five appears to require both “79” on the distractor sign and “446” on the train. The sign therefore contributes necessary evidence to the original question. Removing it or restricting the question to the train region, would change the correct count.
  3. The experiment of content probe does not identify the contribution of target-region representation. The probe combines pooled target-region states with contextualized candidate representations extracted from the AA pass, which may already contain information from the image and question. Although the reported 70.5% accuracy establishes decodability from a joint representation, but does not show how much useful information is provided by the target-region component.
  4. The paper is not self-contained, with some tables in the appendix but the corresponding text description in the main paper.
  5. The experiments only partially separate perceptual saliency from semantic competition. Distractor attenuation changes chroma, contrast, and spatial detail together. These edits may reduce visual prominence, but they may also weaken the recognizability of the distractor-linked answer. For example, changing the color of a competing object can directly alter the attribute that makes an incorrect color option attractive.
Questions for authors
  1. Regarding the example in Figure 1, is the example included in the intervention experiments? How to verify that removal and region-scoped rewriting preserve the correct answer, particularly for counting and compositional questions?
  2. It would be better to supplement the independent-query protocol with an evaluation that incorporate localization and answering within the same inference sequence? How would its results change the interpretation of SFR?
  3. Regarding the probe attribution, what performance is obtained from candidate representations alone? What happens when target-region features are replaced with shuffled features or matched non-target features?
4 comments
AuthorsOpenReview ↗

Responses to Reviewer GcQB (Part I)

To reviewer GcQB: We appreciate the careful reading and the constructive questions. Responses to specific comments are given below, and the corresponding revisions are highlighted in blue in the revised paper.

  1. W1 & Q2: Three results now place the localization inside the answering computation.

    (1) Localization and answering in one sequence. Ten models select a TG option and then an AA option in one generation (Table 4(c) in the main text, Table R7 in Appendix K.7). TG in this protocol reaches 81.1% on average.

    Independent queriesOne sequence
    Mean SFR over ten models (%)52.349.4

    SFR stays close to its original value, and the ranking of the ten models is unchanged. The failure persists right after the model has stated the correct location in its own output.

    (2) Localization in the AA forward pass. The where probe is refitted on the forward pass that produces the AA answer. It identifies the evidence region in 79.5% of the wrong answers and 79.8% of the correct ones (Table R7(b)). The location is encoded during the AA computation whether or not the answer is correct.

    (3) The model's own localization as input. Table 3 of the original submission already returns the model's own correct TG answer together with the AA question (TG fb.). This condition recovers 3.8% of the failures. Supplying the localization in the same context does not repair the answer.

    The independent protocol prevents SG and TG from acting as hints for AA, and result (1) shows that this choice changes SFR only slightly. Results (1) and (2) are added to Sections 4.2.1 and 5.2.2.

  2. W2 & Q1: We agree with the reviewer's reading of the original Figure 1. Its count draws on both the sign and the train, so the sign also carries evidence. Figure 1 is replaced with an item from the failure set of Claude-Opus-4.8. The question asks which metal vessel rests on the cork trivet. The model identifies the black cat as the most salient object and selects the region containing the kettle, yet answers with the frying pan under the cat. Removal and region rewriting both return the kettle, and a location hint does not (Figure 1(a)).

    To verify answer preservation at scale, a stratified sample of 461 items is audited (Table R9, Appendix K.9). Two annotators judge independently whether the gold answer still holds after Removal and after region rewriting, and a third resolves disagreements. Cohen's κ is 0.83 and 0.86.

    RemovalRegion rewrite
    Invalid items, all (%)6.75.6
    Invalid items, counting (%)14.913.8
    Recovery on audited failures (%)62.753.4
    Recovery after excluding invalid items (%)64.854.9

    Counting and Reasoning questions are the most affected, as the reviewer expected. An invalid item is still scored against its original gold answer, so a correct reading of the edited input counts as unrecovered. Excluding the invalid items therefore raises recovery, and Removal + Crop rises from 69.2% to 71.3% in the same way. The recovery reported in Table 3 is thus a conservative estimate. The audit is added to Section 5.1.2.

AuthorsOpenReview ↗

Responses to Reviewer GcQB (Part II)

  1. W3 & Q3: The requested ablations are added to Table 2(b) and Table R10 (Appendix K.10), with the same folds as the original probe. Accuracy averaged over the seven models is 70.5% for the original probe. It falls to 46.9% with candidate representations alone, 46.6% with target states shuffled across questions, and 46.8% with a matched non-target region. The target component therefore supplies the information that lifts the probe from 46.9% to 70.5%, and this information is specific to the target region.

    On the concern about contextualized candidates, Text cand. pairs the target states with candidate representations computed without the image. It still reaches 62.9%. Most of the accuracy is therefore provided by the target region and not by image information inside the candidate representations. These results are added to Section 5.2.2.

  2. W4: We agree that several results were previously reported in the main text only through pointers to the appendix. The revision addresses this in three ways.

    (1) The new results relevant to the main claims now appear in main text tables. Table 1 adds the DL column and a panel with the mean SG, TG, and AA of each task type. Table 2(b) adds two probe ablations, Table 3 adds the Rewrite only control, and the new Table 4(c) reports localization and answering in one sequence.

    (2) Every result in Appendix K is stated in the main text with its key number and a pointer to the full table.

    (3) The main text now uses the full ten pages.

  3. W5: Attenuation is repeated with the chroma scale fixed at 1.0, so the chroma of the distractor is kept and only contrast and blur vary (Table R12, Appendix K.12). The same 318 images and 1,297 questions are used.

    Answer error, L0 to L3 (%)Human distractor preference, L0 to L3 (%)
    Original operator61.4 to 47.973.6 to 48.1
    Chroma fixed61.4 to 53.174.0 to 57.3

    Error still falls by 8.3 points with the chroma of the distractor unchanged, so the effect does not depend on altering the attribute that makes a color option attractive. The preference ratings come from a separate panel of 24 raters. Answer error and human preference fall in about the same proportion under both operators. Error therefore follows how strongly the distractor draws attention. This result is added to Section 4.3.

We hope these results address the reviewer's concerns, and we are happy to discuss any further questions.

Reviewer_GcQBOpenReview ↗

Response for Authors

Thanks for the responses. My follow up question for W1 is that the results can show that the model has correct localization but wrong answer prediction, it does not tell whether it is because the model fail to extract the relevant information or fail to use it? For W5, although the chroma of the distractor is unchanged, blur and lower contrast could also make the distractor less attention-catching or harder to recognize. The experiment does not separate those two effects.

AuthorsOpenReview ↗

Responses to the Follow-up Questions

To reviewer GcQB: We appreciate the follow-up questions. Responses are given below.

  1. W1 (extracting or using the evidence): The internal analyses separate the two. They show that the evidence is extracted into the target-region representation, and that the failure arises when the answer is formed.

    (1) The target-region states carry the answer even when the model answers wrong. The content probe reads the correct answer from the target-region states on 70.5% of the questions the model answers correctly, and also on 70.5% of those it answers wrong (Table 4(b)). This information comes from the target region itself. Without the target states, or with shuffled or matched non-target states, probe accuracy falls to about 47% (Table R10).

    (2) Replacing the target-region states does not repair the answer. Patching uses the model's own Removal run, which answers correctly, as the source. Replacing the target-region states of the failed run with those of this correct run recovers only 25.3% of the failures at the shallowest depth. Replacing the distractor-region states recovers 72.6% (Figure 5(a), Table R11). The target-region states of a failed run therefore already carry most of what the correct run needs, and the failure is driven by the distractor region.

    (3) The readout and the answer fail differently. When the probe errs, 34.0% of its errors select the distractor-linked option, close to the 33.3% expected by chance. The same models choose that option in 48.6% of their own errors (Section 5.2.2). The pull toward the distractor appears in the answer, and not in the extracted evidence.

    The evidence is extracted, and the model fails to use it, as stated in Section 5.2.2.

  2. W5 (attention or recognizability): To separate the two effects, we ran an additional recognition test on the same 1,297 questions from 318 images. At every attenuation level, each of the seven models identifies the distractor object in a four-way question, with chance at 25%. Neutral applies the same edit to an irrelevant region.

    (a) Distractor recognition accuracy (%)

    ModelL0L1L2L3Neutral
    mplug-owl-7B94.193.192.891.894.4
    Qwen3-VL-2B94.493.792.292.194.6
    LLaVA-v1.5-7B94.494.393.693.194.4
    Qwen3-VL-8B95.794.693.893.896.3
    Qwen3-VL-8B-Thinking95.896.095.194.495.5
    InternVL3.5-8B95.895.494.392.995.6
    MolmoPoint-8B96.796.095.495.396.7
    Macro mean95.394.793.993.395.4

    (b) Answer error (%) on the questions whose distractor each model still recognizes at L3

    ModelQuestionsL0L3Reduction
    mplug-owl-7B1,19068.254.313.9
    Qwen3-VL-2B1,19466.252.014.2
    LLaVA-v1.5-7B1,20764.050.713.3
    Qwen3-VL-8B1,21660.447.013.4
    Qwen3-VL-8B-Thinking1,22458.845.912.9
    InternVL3.5-8B1,20555.943.312.6
    MolmoPoint-8B1,23652.640.711.9
    Macro mean1,21060.947.713.2
    All questions (Table 19)1,29761.447.913.5

    Recognition falls by only 2.0 points from L0 to L3, while answer error falls by 13.5 points. The distractor remains recognizable at every level, and the edit on an irrelevant region leaves recognition unchanged. On the questions whose distractor is still recognized at L3, error falls by 13.2 points, close to the full effect of 13.5 points, and every model shows a reduction of at least 11.9 points. Reduced recognizability therefore does not explain the attenuation effect, which follows the drop in attention.

We hope these responses address the follow-up questions, and we are happy to discuss further.

Reviewer_kiS4OpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DECOY, a benchmark designed to evaluate whether multimodal large language models (MLLMs) can effectively utilize task-relevant visual evidence under conflicts between visual saliency and task relevance. The benchmark comprises 2,163 natural images from four eye-tracking datasets and 9,596 human-written question-answer pairs across five task types. The authors use human fixation data to quantify conflict strength through a fixation-based saliency gap and decompose evaluation into three independent dimensions: Saliency Grounding (SG), Target Grounding (TG), and Answer Accuracy (AA). They further define the Suppression Failure Rate (SFR) as the probability of an incorrect AA response conditioned on correct SG and TG responses. Evaluations across 24 MLLMs reveal that models may still be influenced by salient distractors in their final answers despite correctly identifying the most salient object and localizing task-relevant evidence. Even the best-performing model achieves an SFR of 37.7%, substantially higher than the human rate of 4.9%. Answering errors also increase monotonically with the saliency gap. Further experiments involving within-item distractor attenuation and controlled interventions show that removing salient distractors, appending target crops, or restricting the question to the target region is considerably more effective than providing location hints alone. Finally, the authors investigate the underlying mechanisms through content probing with a linear scorer and activation patching. They find that answer-relevant information remains decodable from internal representations, while the causal influence of visual-region states on the final answer diminishes with decoder depth. Overall, the paper reveals a persistent gap between evidence localization and utilization in MLLMs and provides a dedicated benchmark and empirical analyses for investigating this failure mode.

Critical strengths
  1. The research problem is well-motivated, and the benchmark is purposefully designed.

  2. Comprehensive experimental validation with complementary evidence at multiple levels.

Critical weaknesses
  1. The conditional definition of SFR may not fully capture model failure modes under conflicts between visual saliency and task relevance. The paper defines suppression failure as an incorrect Answer Accuracy (AA) response conditioned on correct Saliency Grounding (SG) and Target Grounding (TG). While this design helps isolate failures in evidence utilization after successful grounding, it may exclude other failure modes closely related to the research question. In particular, when a salient distractor causes the model to fail to correctly localize the task-relevant target (TG = 0), this may itself represent an important failure mode arising from the conflict between visual saliency and task relevance, yet such cases are excluded from SFR. Conversely, SG = 0 does not necessarily imply that the model is unaffected by salient distraction, since SG measures agreement with human saliency judgments rather than the model's actual attention allocation. Furthermore, the sets of samples satisfying SG = 1 and TG = 1 may differ across models, potentially introducing conditional sample-selection effects into cross-model SFR comparisons. Although the paper reports Eligibility and partially addresses this concern through an analysis restricted to commonly eligible questions, these evaluations do not yet provide a complete characterization of how models handle saliency conflicts across different grounding outcomes.
Questions for authors

Q1. Does the scope of SFR potentially overlook other failure modes? The paper defines suppression failure under the condition that both SG and TG are correct (SG=1, TG=1). However, conflicts between visual saliency and task relevance may also lead to failures in target grounding (TG=0), while incorrect saliency grounding (SG=0) does not necessarily imply that the model is unaffected by salient distractors. Could the authors report and analyze the AA error rates and the proportions of distractor-linked errors under different SG/TG conditions? This would help clarify whether SFR captures only a subset of failure modes associated with saliency conflicts, and to what extent cross-model differences in SFR are influenced by the selection of conditionally eligible samples. Q2. Could the criteria for defining the task categories be clarified? DECOY covers five types of visual question-answering tasks, but the current categorization appears to mix two dimensions: visual understanding capabilities and application scenarios. For example, Attribute, Retrieval, and Reasoning primarily correspond to different perceptual or reasoning operations, whereas Safety is more closely associated with judgments in a specific application context. Safety-related questions may themselves involve attribute recognition, relational understanding, and commonsense reasoning, leading to potential overlaps with other task categories.

2 comments
AuthorsOpenReview ↗

Responses to Reviewer kiS4

To reviewer kiS4: Thanks for the constructive comments. Responses to specific comments are given below, and the corresponding revisions are highlighted in blue in the revised paper.

  1. W1 & Q1: SFR targets one failure mode: a wrong answer after both the salient object and the evidence have been identified. The other grounding outcomes are now reported alongside it. Table R1 (Appendix K.1) gives the AA error of each model under all four SG and TG outcomes. It also gives the share of errors that select the distractor-linked option. The summary below is added to Section 4.2.1.

    SGTGQuestions (24 models)AA error (%)Distractor-linked share of errors (%)
    ✓✓138,88153.748.6
    ✓✗31,29860.764.1
    ✗✓38,66357.732.4
    ✗✗11,86263.455.0

    The first row is the eligible set on which SFR is computed. Percentages are averaged equally over models. If errors were spread evenly over the three wrong options, the share would be 33.3%.

    (1) TG = 0. We agree that a failed TG also reflects saliency conflict. AA error rises to 60.7% and 63.4% in these two cells. The distractor-linked share rises to 64.1% and 55.0%, compared with 48.6% in the eligible cell. Adding these cases would only raise the failure rate, so SFR gives a conservative estimate of the failures caused by salience.

    (2) SG = 0. When SG fails but TG succeeds, the distractor-linked share is 32.4%. This is the 33.3% expected by chance. Once the saliency judgment of a model departs from human fixations, its errors are no longer drawn toward the annotated distractor. Conditioning on SG therefore keeps the questions on which the annotated conflict also acts on the model.

    (3) Selection effects across models. The ranking of the 24 models by SFR matches their ranking by unconditional error on all 9,196 incongruent questions, with Spearman ρ = 0.988 (Section 4.2.1, Table R1). Appendix E.2 of the original submission also restricts SFR to the 1,327 questions eligible for all 24 models. SFR changes by 1.4 points on average and by at most 2.6 points, and the ranking correlation is ρ = 0.94. The differences in SFR between models therefore do not come from their different eligible sets.

  2. Q2: All five types follow one criterion: the evidence the target supplies and how the salient distractor competes with it. In HOI and Safety questions, the decisive detail is the object of an interaction or a hazard. Most fixations go to the actor, tool, or machinery that dominates the scene. Retrieval questions ask whether an entity is present, which requires search beyond the distractor. Attribute questions ask for a property on which the distractor carries a competing value. Reasoning questions spread the evidence over several regions. Safety is thus defined by the role of its evidence, a hazard detail away from the salient agent, and not by its application context.

    Most questions engage several operations, so a Safety question can also involve attribute recognition. Each image therefore receives one primary type according to the role of its target. This label reaches Fleiss' κ = 0.84 before adjudication (Table 8). Task type and saliency gap are recorded as separate variables, and the main findings hold within every type. Attenuation reduces error in all five types (Table R4), and the mean SG, TG, and AA of each type are now reported in the lower panel of Table 1. The criterion is added to Section 3.3.

We hope these results address the reviewer's concerns, and we are happy to discuss any further questions.

AuthorsOpenReview ↗

Kind Reminder of the Discussion Period

Dear Reviewer kiS4,

Thank you again for your time and constructive review. As the author-reviewer discussion period will end in less than two days (November 18, AoE), we would like to kindly ask whether our responses have addressed your concerns. AA error and the distractor-linked share of errors are now reported under all four SG and TG outcomes (Section 4.2.1, Table R1), and the criterion behind the five task types is stated in Section 3.3. The revised paper has been uploaded with all changes highlighted in blue. If our responses have resolved your concerns, we would appreciate it if you could take them into account in your final assessment. We are glad to clarify any remaining questions before the discussion closes.

Best regards, The Authors

Reviewer_LfNyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Decoy, a benchmark for diagnosing whether multimodal LLMs use evidence they have located when a more salient distractor competes with it. It pairs 2,163 images from four eye-tracking corpora with 9,596 human-written four-way questions in five task types and computes a fixation-based saliency gap before model evaluation. Each question is scored on Saliency Grounding, Target Grounding, and Answer Accuracy, and a suppression failure is a wrong answer conditional on correct grounding in the first two. The authors evaluate 24 MLLMs and a 26-participant human sample, compare eleven interventions on fixed failure sets, and probe and patch seven open-weight models. They report a lowest suppression failure rate of 37.7% against 4.9% for humans, more recovery from scoping than from hints, and decodable content alongside a depth-dependent decline in distractor-region patching.

Critical strengths
  1. The benchmark is carefully built. The saliency gap is computed from fixations before model evaluation, labeling uses three independent annotators plus two adjudicators for disputed items, agreement is high (Fleiss' κ 0.82 to 0.89, box IoU 0.76 to 0.81), and 400 congruent controls on matched photographs show a 44.1-point error difference (10.7% versus 54.8%, interval [39.6, 48.5]). The same scenes support much higher accuracy when salience aligns.
  2. The within-item attenuation experiment is the strongest causal evidence. On 1,297 fixed questions from 318 images, reducing the distractor's chroma, contrast, and detail lowers error from 61.4% to 47.9% across seven models, while the same edit to an irrelevant region leaves error at 61.2%; distractor preference falls from 73.6% to 48.1% in the 24-rater check. Removal recovers 62.6% of failures against 2.1% for control inpainting, so distractor prominence clearly shapes the answers.
Critical weaknesses
  1. The suppression label covers more than suppression. SFR counts any wrong AA answer after correct SG and TG, but among the seven probe models 48.6% of incorrect answers select the distractor-linked option, against 33.3% chance (Section 4.3), and removal leaves 37.4% of the original failures wrong (Table 3, 62.6% recovery). The 37.7% to 64% range therefore includes errors beyond salience, yet the abstract attributes the failures to the salient distractor. Saliency competition is the benchmark's stated identity.

  2. The mechanistic claim is not identified. The content probe mixes pooled target states with each candidate's own representation (Section 5.2.1, Appendix I.1), with no candidate-only or target-only ablation reported. The depth decline is also what one expects if distractor information is integrated early: a deep patch cannot undo influence already in the residual stream, and the 94.5% answer-token result only shows the output remains writable. The abstract states that visual-region causal influence fades, but what is measured is the efficacy of a particular patch site at a particular depth.

Questions for authors
  1. Can you report a stricter salience-specific metric on the same eligible questions, for example the rate of wrong AA answers that select the distractor-linked option, or the rate that removal repairs? If that rate is near 20% rather than 37.7%, the headline and the human comparison should be restated. This question most affects the rating.
  2. What is the pixel-area distribution of the target boxes, and how do SFR and unconditional error vary with target area? Image plus crop recovers 50.0% of failures while removal recovers 62.6% (Table 3), so part of the effect could be resolution rather than competition. Persistence of the failure for large targets would strengthen the saliency account.
  3. What does the content probe score with candidate features only, with target states only, or with target states permuted across questions? If candidate-only features already reach 70.5%, the target state is not necessary for the readout, and the probe may be reading answer-text compatibility rather than target evidence. The paper reports a permuted-label control (25.4%) but not these ablations.
  4. The target-region curve in Figure 5(a) also declines, though far less than the distractor curve; does the same hold when all visual tokens are patched, and is the distractor-versus-target gap statistically separable? If every visual patch declines, the result concerns the decreasing efficacy of visual patching rather than a distractor-specific route; if the gap remains, the route interpretation is better supported.
  5. What does text-only GPT-5.6 Sol score on SG and TG? Only AA is reported (27.3%, Appendix D.1). Above-chance SG or TG would mean the eligibility set includes items that can be grounded from language priors, which would change the denominator of SFR.
  6. How were the 318 images and 1,297 questions in the attenuation experiment selected? Do they differ from the full benchmark in saliency gap, target area, task mix, or model error? The 13.5-point effect is the paper's cleanest causal estimate, so its selection rule matters.
  7. Is there a control for region-anchored rewriting that rephrases the question without naming the target region? The rewrite changes both the scope and the wording (Appendix H.4), and its 53.5% recovery is the main textual evidence for the scoping account. A rewrite-only control would separate scope from simplification.
  8. Is the 350-item human sample balanced across the five task types? Model SFR is a count-weighted rate over each model's eligible subset of the 9,196 incongruent questions, while the human SFR is computed over a stratified sample whose task composition is not reported; a task-matched human estimate would make the 4.9% versus 37.7% comparison more direct.
4 comments
AuthorsOpenReview ↗

Responses to Reviewer LfNy (Part I)

To reviewer LfNy: We thank the reviewer for the detailed review and the precise questions. Responses to specific comments are given below, and the corresponding revisions are highlighted in blue in the revised paper.

  1. W1 & Q1: We agree that SFR counts every wrong answer after correct grounding, and that not every such error comes from the distractor. Two stricter rates are now reported on the same eligible questions (Table R2, Appendix K.2). DL is the percentage of eligible questions answered with the distractor-linked option. Rem. is the percentage of eligible questions that fail on the original image and are answered correctly after distractor removal. SFR, DL, and Rem. share one denominator, and human responses are scored with the same definitions.

    SFR (%)DL (%)Rem. (%)
    Human4.91.9N/A
    GPT-537.716.927.5
    Gemini-3.1-Pro41.618.829.3
    Claude-Opus-4.844.620.629.3

    (1) The headline is restated. DL is the stricter of the two rates, and its lowest value is 16.9% for GPT-5. The abstract and the introduction now report this value against 1.9% for humans. DL is added as a column of Table 1 for all 24 models, and its definition is added to Section 3.3. The human comparison holds under both rates. The best model fails about eight times as often as humans under SFR and about nine times as often under DL (Section 4.2.1).

    (2) DL and SFR bound the failures caused by salience on the eligible questions. DL counts only the errors that take their content from the distractor. A salient distractor can also cause errors that land on other options, so DL is a lower bound on the failures caused by salience. SFR also contains recognition and reasoning errors, as stated in Section 3.3, so it is an upper bound. Models stay far from humans at both ends.

    (3) Removal repairs most of each model's SFR. Across the ten intervention models, Rem. lies between 27.5% and 35.7% of the eligible questions. For GPT-5, removal repairs 27.5 of its 37.7 points, while inpainting a matched control region repairs 2.1% of the failures on average (Table 3). Within the eligible errors, 48.6% select the distractor-linked option, compared with 33.3% if errors were spread evenly over the three false options (Table R1). As the reviewer notes, distractor prominence clearly shapes the answers. The restated headline now reports the part of the failures that is traced to the distractor directly.

AuthorsOpenReview ↗

Responses to Reviewer LfNy (Part II)

  1. Q2: Table R3 (Appendix K.3) groups the 9,196 incongruent questions into quintiles of relative target area. The median target grows from 1.8 to 53.8 thousand pixels between A1 and A5.

    Area quintileA1A2A3A4A5
    Relative area (%)[0.1, 0.7](0.7, 1.6](1.6, 3.1](3.1, 6.2](6.2, 32.6]
    SFR (%)70.661.357.347.239.1
    Unconditional error (%)71.262.358.147.939.6

    Small targets are harder, consistent with the size effect of Zhang et al. (2025a). The failure persists for large targets, with SFR still at 39.1% in A5. Within A5, unconditional error rises from 31.7% to 50.7% across the saliency gap quintiles. Target size therefore does not account for the gradient. The intervention results point the same way. Region rewriting adds no pixels and recovers 53.5% of the failures, slightly more than the 50.0% of Image + Crop (Table 3). These results are added to Section 4.3.

  2. Q5: With the image withheld, GPT-5.6 Sol reaches 27.9% on SG and 35.8% on TG, against 25% chance (Table R6, Appendix K.6). Since TG is above chance, every question whose TG the text-only run answers correctly is removed from the eligible set of each model. SFR then rises by 1.0 point on average, and GPT-5 moves from 37.7% to 39.6%. Questions that can be grounded from language priors therefore lower SFR slightly, and the reported SFR is a conservative estimate. This result is added to Section 4.2.1.

  3. Q6: The attenuation images are selected before any model is run (Appendix K.4). Quotas are allocated to the four corpora in proportion to their image counts, and 400 images are drawn by stratified random sampling with a fixed random seed. Each image is screened by two checkers, and a third resolves disagreements. Of the 400 images, 47 are excluded because the edit would enter task evidence, 21 because the answer could not be preserved, and 14 because no matched Neutral region exists. The remaining 318 images provide 1,297 questions. Model answers, error rates, and human preference play no role in the selection.

    Mean ΔMedian target area (%)Original error (%)
    Attenuation subset0.3712.161.4
    All incongruent questions0.3462.360.1

    The task shares of the subset differ from those of the full benchmark by at most 0.4 points (Table R4(a)). Error falls from L0 to L3 in all five task types (Table R4(b)). The selection rule and this comparison are added to Section 4.3.

  4. Q8: The 350 human items contain 292 incongruent items, and their task shares match those of DECOY within 0.3 points, for example 26.4% versus 26.3% for HOI (Table R5, Appendix K.5). Human SFR is 4.9% [3.0, 7.5] on 824 eligible responses. After the task estimates are weighted by the DECOY shares, it remains 4.9% [2.4, 8.0]. The reweighted estimate is added to Section 4.2.1.

AuthorsOpenReview ↗

Responses to Reviewer LfNy (Part III)

  1. W2 & Q3: The requested probe ablations are added to Table 2(b) and Table R10 (Appendix K.10). They use the same folds as the original probe, and accuracy is averaged over the seven models.

    OriginalCand. onlyText cand.ShuffledNon-target
    70.546.962.946.646.8

    (1) The target states are necessary. Candidate features alone reach 46.9%, far below the 70.5% of the full probe.

    (2) The readout depends on what the target region contains. Shuffling the target states across questions gives 46.6%, and replacing them with a matched non-target region gives 46.8%. Both fall to the level of candidate features alone.

    (3) Most of the accuracy comes from the target component. Text cand. pairs the target states with candidate representations computed without the image. It still reaches 62.9%, so the probe does not rely on image information carried by the candidates.

    These ablations are added to Section 5.2.2.

  2. W2 & Q4: We agree with the reviewer's reading of the depth result. Table R11 (Appendix K.11) adds All visual patches on the same 12,604 model and question pairs. Recovery (%) at selected depths is shown below.

    Depth0.150.450.750.850.95
    Distractor72.667.930.416.28.5
    Target25.322.419.117.815.9
    All visual94.188.349.728.318.6
    Distractor minus target [95% CI]47.3 [44.1, 50.6]45.5 [42.3, 48.8]11.3 [8.2, 14.4]−1.6 [−4.3, 1.1]−7.4 [−9.7, −5.1]

    Every visual patch loses efficacy with depth, including All visual. Distractor patches exceed target patches through depth 0.75, with intervals above zero. The two sites do not differ at 0.85, and target patches recover more at 0.95. The depth result therefore concerns the efficacy of patching visual states, as the reviewer suggests. The abstract, the contributions, Section 5.2, and the conclusion now state it in these terms, and the sentence on a distractor route is removed from Section 5.2.3. The 94.5% answer token result is used only to show that the answer remains modifiable. The comparison above is added to Section 5.2.3.

  3. Q7: Table 3 now includes a Rewrite only control. It rephrases each question with matched length and difficulty, keeps the original scope, and gives no information about the target region. On the same failure sets, it recovers 4.6% of the failures, compared with 53.5% for region rewriting (Table R8, Appendix K.8). The recovery therefore comes from narrowing the scope of the question. This result is added to Section 5.1.2.

We hope these results address the reviewer's concerns, and we are happy to discuss any further questions.

AuthorsOpenReview ↗

Kind Reminder of the Discussion Period

Dear Reviewer LfNy,

Thank you again for your time and detailed review. As the author-reviewer discussion period will end in less than two days (November 18, AoE), we would like to kindly ask whether our responses have addressed your concerns. For Q1, which you noted most affects the rating, the distractor-linked rate is now reported on the same eligible questions. Its lowest value is 16.9%, against 1.9% for humans, and the abstract and the introduction now report this comparison (Table 1, Table R2). The probe ablations and the comparison with patching all visual tokens are added to Sections 5.2.2 and 5.2.3. The revised paper has been uploaded with all changes highlighted in blue. If our responses have resolved your concerns, we would appreciate it if you could take them into account in your final assessment. We are glad to clarify any remaining questions before the discussion closes.

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Our Responses & Paper Revisions

We sincerely thank all reviewers for their careful and constructive reviews. We appreciate that the reviewers found the research problem well motivated and important (Reviewers kiS4, GcQB) and the benchmark carefully built and substantial (Reviewers LfNy, GcQB, kiS4). They also noted that the saliency gap is computed from human fixations independently of the evaluated models (Reviewers LfNy, GcQB). In addition, they found the within-item attenuation experiment to be strong evidence that the competing region shapes model answers (Reviewers LfNy, GcQB), and the validation comprehensive with complementary evidence at multiple levels (Reviewer kiS4).

We have made our best effort to address the concerns and revise the paper accordingly. The major modifications are summarized as follows.

  1. Stricter failure rates and the headline (Reviewers LfNy, kiS4): The distractor-linked rate (DL) is defined in Section 3.3 and added to Table 1 for all 24 models. The abstract and the introduction now report the lowest DL of 16.9% against 1.9% for humans. AA error and the distractor-linked share of errors are reported under all four SG and TG outcomes (Section 4.2.1, Table R1). DL and the rate repaired by removal are reported on the same eligible questions as SFR (Table R2).

  2. Localization within the answering sequence (Reviewer GcQB): The new Table 4(c) reports SFR when TG and AA are answered in one generation. A probe on the AA forward pass locates the evidence equally well for wrong and correct answers (Sections 4.2.1 and 5.2.2, Table R7).

  3. Validity of the interventions (Reviewers GcQB, LfNy): Figure 1 is replaced with an item from the intervention set. A stratified audit of 461 items verifies that Removal and region rewriting preserve the gold answer (Section 5.1.2, Table R9). A Rewrite only control is added to Table 3 and separates scope from rewording (Table R8).

  4. Probe and patching analyses (Reviewers LfNy, GcQB): Probe ablations with candidate features alone, candidates computed without the image, shuffled target states, and matched non-target regions are added to Table 2(b) and Table R10. Patching all visual tokens is compared with distractor and target patches, with intervals for their difference (Table R11). The depth result is stated as the efficacy of patching visual states in the abstract, the contributions, Section 5.2, and the conclusion.

  5. Controls on stimuli and samples (Reviewers LfNy, GcQB): We add four controls. (1) Failure rates across target sizes and across saliency gaps within large targets (Section 4.3, Table R3). (2) The selection rule and representativeness of the attenuation subset, with the effect in each task type (Section 4.3, Table R4). (3) Attenuation with the chroma scale fixed (Section 4.3, Table R12). (4) Text-only SG and TG, and a human SFR reweighted by task (Section 4.2.1, Tables R5 and R6).

  6. Task definition and presentation (Reviewers kiS4, GcQB): The criterion behind the five task types is stated in Section 3.3. The mean SG, TG, and AA of each task type are given in the lower panel of Table 1. Every new result is stated in the main text with its key number and a pointer to Appendix K.

All new results are collected in Appendix K (Tables R1 to R12). The major revised contents in the manuscript are highlighted in blue. Responses to specific comments are given in the reply to each reviewer. We welcome any further discussion and will address any remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.