acceptodds
Under review as a conference paper at ICLR 2027

Change the Image, Change the Answer? Predicting VLM Responses to Regional Edits

Abstract

A vision–language model that answers an image question correctly may follow a regional edit intended to change the answer-relevant facts, or retain its original answer. We ask whether original-image measurements can rank these responses in multiple-choice VQA without observing the edited image. We propose GLANCE, a three-view score fixed before evaluation contrasting original-answer support on the full image and an enlarged regional crop with support remaining after masking. Edited images supply labels and determine admission to the primary cohorts but never enter the score. On six model-specific Visual7W cohorts selected by post-edit checks, GLANCE ranks target-following above preservation with AUROC 0.707–0.832 and exceeds the strongest tested attention and geometry readouts in five. On two models, GLANCE also ranks changes among originally correct answers without applying crop or placebo admission to the new edits. In matched diagnostic probes, adding masked support to visible supports raises AUROC by 0.124–0.243 across six Visual7W and two single-layer MEGA cohorts, with all refitting-sensitivity intervals above zero. Positive increments also persist in two small subsets where a majority of human raters selects the target answer. Component analyses show that masked-view access matters more than the exact three-view weighting; the tested binary answer-agreement rules generally retain less ranking information than continuous support. The score ranks response tendencies under the tested edits; among changed answers, it does not consistently separate target from third-option responses.

Primary area:
interpretability and explainable AI
Keywords:
vision-language models, visual question answering, counterfactual evaluation, response prediction, evidence removal, interpretability

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_QSjYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether the responses of commonly used vision-language models (VLMs) to regional image edits can be predicted using measurements obtained solely from the original image. The authors propose GLANCE, a training-free method that can serve as a tool to assess how VLMs are affected by such edits. The method uses three types of inputs to compute its score: the original image, a cropped image of the target region, and an image with the target region masked. GLANCE provides a score that indicates whether a VLM is more likely to follow the edit or preserve its original answer. Experiments show that GLANCE outperforms other observation methods based on attention, geometry, and related features. The authors also evaluate the robustness of the method using alternative masking strategies, editing seeds, and human validity judgments. Overall, this paper investigates a novel research problem. Compared with previous methods, GLANCE makes it possible to predict a model's sensitivity to visual evidence before an edit is performed. However, I still have several concerns.

Critical strengths
  1. The paper moves beyond existing post-edit evaluation methods and instead investigates the problem from the perspective of predicting model behavior.

  2. The paper evaluates a wide range of mainstream VLMs, and the experimental results are convincing.

  3. The paper provides insightful and interesting findings. For example, in Fig.3, masked-region support contributes substantial predictive information beyond visible-image measurements.

  4. The discussion is relatively comprehensive and includes the limitations of the proposed approach.

Critical weaknesses
  1. The paper explains how the proposed method differs from previous approaches and provides extensive experiments to demonstrate its effectiveness and robustness, but I am still unclear about the specific practical significance or motivation. The paper does not seem to explain this explicitly.

  2. The paper only provides edits using two different seeds, but it does not seem to clearly specify or strictly control the extent of the edits. For example, different edits may lead to different model responses. If an image is edited from a giraffe to a moose, the model may follow the edit. However, if the edit changes a moose to an elk, which has very similar visual features, the model may preserve its original answer.

  3. The advantage of the proposed GLANCE formulation over simpler alternatives is not established. In Table 1, the results do not clearly demonstrate the advantage of GLANCE over the crop-minus-mask baseline.

Questions for authors
  1. Could the authors provide a more clear motivation and some concrete practical examples of why we need GLANCE?

  2. Could the authors provide a clearer explanation of the editing process and explain will GLANCE show different performance when dealing with difference levels of editions (for example, moose -> elk and moose -> giraffe)?

  3. Could the authors explain why GLANCE is better than the crop-minus-mask baseline?

5 comments
AuthorsOpenReview ↗

Response to QSjY (1/2): Practical Use, Cost, and Edit Similarity

Thank you for recognizing the pre-edit prediction problem and the masked-support finding. We address your three questions below.

Q1. Practical use and cost

A concrete use is estimating a model's answer-change rate over a collection of regional edits. A practitioner scores the original items, forms strata, then allocates editing and verification across those strata. Sampling weights recover the population estimate. The benefit is fewer edited and labeled items for the same estimation precision.

On 500 new-source items, the originally correct sampling frames contained 449/455 items for 7B/3B. Each design sampled 100 items per model, with scores and sampling lists frozen before editing.

Prospective resultQwen2.5-VL-7BQwen2.5-VL-3B
T2 AUROC0.7570.751
GLANCE: reduction in required edits/labels at matched precision13.1%12.6%
Single-view −g_mask: same criterion18.9%15.8%
GLANCE break-even time per rater-item, three raters15.3 s16.7 s
Single-view break-even time per rater-item2.0 s3.1 s

Completing the remaining items established the full-pool reference for evaluating coverage and comparing the frozen design with simple random sampling at matched precision. Four of six initial intervals covered the full-pool value; subsequent design simulations had approximately 95% coverage. Seven-model retrospective results similarly reduce requirements by 10.6%–19.9%, compared with −0.22% to +1.44% for confidence or visible-only stratification under the same design.

The cost analysis combines measured machine costs with timing from the separate human audit. Machine-only self-scoring costs more than direct labeling; with a 7B reference scorer, three-view scoring adds cost and single-view scoring approximately breaks even when edits must be generated. For human verification, the measured median of 10.01 s per rater-item exceeds the single-view break-even thresholds and falls below the three-view thresholds. This cost model supports single-view scoring for audits requiring human verification.

Using a fixed 7B reference scorer for 32B/27B at n=100 per design reduces matched-precision label requirements by 12.40% / 12.06% with GLANCE and 13.51% / 15.31% with −g_mask. Lists were frozen before target-model responses on existing edited assets; five of six intervals covered the full-pool value. These reductions require no target-model three-view scoring. The revised Introduction and Section 4.8 describe this audit workflow and its break-even conditions (Table 4; Appendix J.4, Tables 59–61).

Q2. Editing process and near versus far targets

The pipeline inpaints region R toward a target answer while retaining pixels outside R. Different seeds vary the realized edit; target swaps vary semantic distance. We therefore tested your moose-to-elk versus moose-to-giraffe concern with both stratification and a paired target-swap experiment.

TestResult
Existing targets: semantically closest thirdT1 AUROC 0.76–0.86; T2 AUROC 0.695–0.783
Existing edits: high versus low pixel change inside RAll AUROC-difference intervals include zero
Same original image, question and region; near versus far targets, six modelsNear-minus-far follow rate +0.032 to +0.072; only Gemma's interval excludes zero
Same paired experimentNear-minus-far T2 AUROC +0.012 to +0.078; only Llama's interval excludes zero

All six paired AUROC point estimates favor near targets. In the observational T2 strata, the direction varies by model, including −0.055 for LLaVA-OneVision and +0.054 for 3B. Prediction remains informative for close targets, with model-dependent differences across the observational strata.

Near/far is defined by text similarity. A post-result, single-author check of 40 pairs found distinct targets in 38/40 and judged 32/40 near versus 25/40 far targets realizable inside R alone. Near targets were thus more often judged realizable within the original region, providing descriptive context for the paired AUROC results. The editing procedure is detailed in Appendix C.1; Section 4.7 and Appendix I.7 report both analyses, with paired results in Table 57.

AuthorsOpenReview ↗

Response to QSjY (2/2): Three-View and Single-View Readouts

Q3. Why use the three-view combination?

GLANCE averages full−mask and crop−mask. Full and crop measure support in different viewing contexts, so combining them has an empirical, endpoint-dependent value. In the original six T1 cohorts, GLANCE and crop−mask have paired AUROC differences of −0.011 to +0.010, with every interval including zero. On new-source 3B T2, a direct paired comparison favors GLANCE by +0.017 [+0.001, +0.032].

The initial seven-model T1/T2 panel (14 cells, evaluated before the Pixtral/Aya extension, with −g_mask as a post-hoc single-view comparator) measures each readout's maximum gap from each cell's best readout:

ReadoutMaximum AUROC gap [95% CI]GLANCE minus alternative gap [95% CI]
GLANCE0.0251 [0.0189, 0.0419]—
crop−mask0.0349 [0.0289, 0.0537]−0.0099 [−0.0219, −0.0020]
full−mask0.0418 [0.0343, 0.0801]−0.0168 [−0.0504, −0.0055]
−g_mask0.0623 [0.0457, 0.0983]−0.0372 [−0.0720, −0.0098]

GLANCE has the smallest maximum gap in this 14-cell panel, with paired intervals favoring it over all three alternatives. In the expanded 24-cell sensitivity panel, the intervals for GLANCE's maximum-gap differences from full−mask and −g_mask include zero. On nine-model T2, −g_mask significantly exceeds GLANCE on 7B, 3B, 32B, Llama and Pixtral.

This measured tradeoff clarifies the use of each readout: the single masked view is a strong, lower-cost choice for T2 answer-change prediction, while GLANCE provides a balanced fixed combination for evaluation across T1 and T2. The revised Introduction, Sections 3.2 and 4.3, and Appendix E.6 (Table 26) explain the predictive finding and these complementary uses.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer QSjY,

Thank you for asking when a practitioner would use GLANCE. Table 4 gives a concrete example: when estimating how often a model changes its answer after an edit, GLANCE reduced the number of edits and labels needed for the same precision by 13.1% on the 7B model and 12.6% on the 3B model. Does this example address your question about practical u

Reviewer_cweSOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether measurements on the original image can predict how a VLM will respond to a later regional edit in multiple-choice VQA. The proposed GLANCE score combines original-answer support on the full image, an enlarged crop and a gray-masked version of the region.

Critical strengths
  1. The controls and statistics are extensive (geometry matching, displaced mask, mask-fill ablation, image-grouped bootstrap intervals).
  2. Negative results (T3, answer selection, GRPO reward) are reported honestly, and a replay archive is provided.
Critical weaknesses
  1. Significance is my main concern. Masking R and inpainting R are two interventions on the same pixels, so it is expected that one predicts the other. Practical value is also not shown: selecting answers by GLANCE lowers accuracy (0.837 vs. 0.920 on 7B), the GRPO reward gives no consistent gain, and triage is retrospective. I suggest the authors show a concrete case where pre-edit prediction is needed and cheaper than just running the edit.
  2. The preserve label is confounded with edit failure, and the main population is selected with post-edit information. Among invalid edits in the human audit, 17/20 and 18/20 are preserve items, GLANCE cannot separate valid from invalid preserve (0.542/0.553), and agreement is only moderate (Fleiss' kappa 0.424/0.496) on two models. T1 also keeps only items that already follow the edit on the crop (e.g., 568 of 2,073 for Qwen3.6-27B). A larger human validation and the admission-free T2 setting for all six models are needed.
  3. Generality is limited. There is one dataset, one editor, and five of six models use Qwen-family backbones. The only non-Qwen model (Gemma-3-12B) is the weakest (0.707) and is not separated from geometry. The score also needs a ground-truth box, and with predicted boxes AUROC drops by about 0.08-0.17.
Questions for authors

Same as weaknesses.

4 comments
AuthorsOpenReview ↗

Response to cweS (1/3): Predictive Information and Prospective Auditing

Thank you for the concrete requests on significance, label validity and generality. We address each with the corresponding evidence.

C1. Information beyond intervening on the same pixels

Masking removes evidence; generative editing introduces a proposed alternative. Using the same views and region, we compare which pre-edit measurements predict the later response.

ComparisonResult
Continuous GLANCE versus binary pixel-FPVGAUROC advantage +0.1395 to +0.1719 in seven of eight pools; paired intervals exclude zero. Gemma: +0.0295 [−0.0091, +0.0684].
Add masked support to matched full/crop probesAUROC increment +0.1243 to +0.2426 across eight cohorts; all refitting-sensitivity intervals are positive.
Add masked support after full/crop, geometry and blank-image controlsPositive OOF AUROC increments in all eight cohorts, including Gemma +0.0493 [+0.0191, +0.0794].

The eight-cohort panel comprises six Visual7W and two mixed-source cohorts. The fixed-formula comparisons evaluate GLANCE's combined score; these out-of-fold probes quantify the predictive information added by the masked view. Geometry/blank-control intervals condition on fixed out-of-fold predictions.

An exploratory same-edit analysis holds the actual edit fixed. Across 2,724 discordant model pairs on 627 items, the model with higher within-model GLANCE percentile changes its answer in 0.6278 [0.6003, 0.6544] of pairs; −g_mask reaches 0.6993 [0.6742, 0.7248]. Under a common reference distribution, −g_mask exceeds reverse full-image support by +0.0762 [+0.0442, +0.1087], while GLANCE's difference is +0.0084 [−0.0314, +0.0479]. Controlling model-pair base rates in an out-of-fold diagnostic reduces log-loss by 0.0490 [0.0286, 0.0682] with GLANCE and 0.1116 [0.0832, 0.1389] with −g_mask. These diagnostic intervals condition on the fitted out-of-fold predictions.

Together, these controls establish predictive information in continuous masked support beyond the tested binary rules, visible support and region properties, including information about model responses to identical edits. The revised Introduction and Sections 4.3–4.4 organize the contribution around this result (Figures 3–4; Appendix E.5, Table 25).

C2. A prospective use and when it is cheaper

The concrete use is stratified estimation of a population's answer-change rate. On 500 new-source items, scores and sampling lists were frozen before generating the selected edits (originally correct frames N=449/455; n=100 per design). After completing the pool for validation, GLANCE required 13.1% / 12.6% fewer edited and labeled items at matched precision for 7B/3B; single-view −g_mask required 18.9% / 15.8% fewer. Four of six initial intervals covered the full-pool value; design simulations gave approximately 95% coverage.

Cost condition, three raters7B3B
GLANCE break-even verification time per rater-item15.3 s16.7 s
Single-view −g_mask break-even time2.0 s3.1 s
Median verification time in the separate human audit10.01 s10.01 s

The cost analysis combines measured machine costs with timing from the separate human audit. Machine-only self-scoring costs more than direct labeling; with a 7B reference scorer, three-view scoring adds cost and single-view scoring approximately breaks even when edits must be generated. For human verification, the measured 10.01 s exceeds the single-view thresholds and falls below the three-view thresholds. This supports the masked-support component for human-verified audits under the measured cost model.

A fixed 7B reference scorer reduces matched-precision label requirements for 32B/27B at n=100 per design by 12.40% / 12.06% with GLANCE and 13.51% / 15.31% with −g_mask. Those lists were frozen before target-model responses on existing edits; five of six intervals covered the full-pool value. These reductions require no target-model three-view scoring. Section 4.8 and Table 4 present this application and its break-even conditions; Appendix J.4 gives the full results (Tables 59–61).

AuthorsOpenReview ↗

Response to cweS (2/3): Answer-Change Prediction and Human Validation

C3. T2 without post-edit admission, across nine models

We have added T2 for every evaluated model. T2 keeps originally correct items with a parseable edited full-image answer, removes edited-crop and placebo admission, and counts any answer change, including a third option.

ModelGLANCE T2 AUROC
Qwen2.5-VL-7B0.7629
Qwen2.5-VL-3B0.7484
LLaVA-OneVision0.7773
Qwen3-VL-32B0.7524
Gemma-3-12B0.6928
Qwen3.6-27B0.7850
Llama-3.2-11B-Vision0.7304
Pixtral-12B0.7813
Aya-Vision-8B0.7790

All intervals lie above 0.5; 32B and 27B start from fixed 1,000-item subsamples, with 999 parsed responses for 32B. The single masked view significantly exceeds GLANCE on 7B, 3B, 32B, Llama and Pixtral.

The original T1 asks a different, conditional question: follow versus preserve after the model answers the target on the edited crop and remains stable on the placebo. For Qwen3.6-27B, its funnel is 2,234 candidates → 2,073 originally correct → 642 crop-target responses → 600 placebo-stable → 568 determinate follow/preserve responses. Section 3.1 defines the populations, Section 4.1 reports the admission funnel, and Table 2A presents nine-model T2; Appendix G.1 gives the full table (Table 31).

C4. Edit validity and the expanded human audit

We completed three-rater judgments on 200 shared edits across five cohorts. Strict validity requires a majority to jointly judge that the target is true, the original answer is false, and necessary information remains visible.

The sampling frame excludes earlier rated edits and oversamples assets preserved by any cohort. AUROC uses inverse-inclusion weights; preserves share one inclusion probability, so their invalidity proportions estimate this frame. Failure of strict validity includes ambiguous or non-majority judgments; proportions from the earlier audits are reported separately.

Raters were blind to model responses, labels and scores; one was an author aware of the hypothesis. Before analysis, 134 author entries affected by a collection-file duplication were excluded and rerated using the frozen interface; the author reported not viewing others' ratings. Fleiss' kappa / pairwise agreement / PABAK are 0.6328 / 0.8200 / 0.6400 for strict validity and 0.6518 / 0.8300 / 0.6600 for target choice.

CohortPreserves failing strict validityValid follow / preserveWeighted valid T1 AUROC [95% CI]
Qwen2.5-VL-7B22/48 (45.83%)56 / 260.7459 [0.6122, 0.8614]
LLaVA-OneVision21/43 (48.84%)53 / 220.7599 [0.5945, 0.8953]
Qwen3-VL-32B33/52 (63.46%)65 / 190.7057 [0.5382, 0.8547]
Gemma-3-12B30/63 (47.62%)33 / 330.7177 [0.5842, 0.8434]
Qwen3.6-27B29/47 (61.70%)52 / 180.7483 [0.6125, 0.8690]

The unweighted 32B sensitivity interval includes 0.5. Within preserves, GLANCE's valid-versus-invalid AUROC is 0.4740–0.5441, with all intervals including 0.5. Human edit validity and the model's observed answer preservation are therefore reported separately.

Under the primary weighted analysis, strict-valid T1 AUROC is 0.7057–0.7599, with every cohort's interval above 0.5. Valid-minus-all-sampled differences range from −0.0327 to +0.0087, with all intervals including zero. GLANCE thus distinguishes follow from preserve even within human-validated edits. Section 4.5 and Table 2B report these results together; Appendix C.3 gives the audit design and Appendix G.2 the full results (Tables 37–38).

AuthorsOpenReview ↗

Response to cweS (3/3): Generality Across Models, Editors, and Regions

C5. Model, dataset and editor coverage

The evaluation now covers nine models, including four non-Qwen backbones. Llama, Pixtral and Aya each separate GLANCE from the strongest tested geometry readout:

Non-Qwen modelT1 AUROCGLANCE minus strongest geometry [95% CI]
Gemma-3-12B0.7068+0.0333 [−0.0325, +0.0957]
Llama-3.2-11B-Vision0.7182+0.0953 [+0.0362, +0.1475]
Pixtral-12B0.7944+0.1434 [+0.0863, +0.1972]
Aya-Vision-8B0.8161+0.1067 [+0.0487, +0.1636]

Two complementary analyses examine Gemma beyond its full-cohort AUROC comparison (+0.0333 [−0.0325, +0.0957]): within area/centrality/aspect-matched strata, GLANCE exceeds centrality in within-stratum concordance by +0.1008 [+0.0128, +0.1918]; adding masked support to the full/crop, geometry and blank-image probe improves OOF AUROC by +0.0493 [+0.0191, +0.0794]. Both identify predictive information beyond measured region geometry.

We retain the recorded scoring configuration. Single-BOS rescoring with labels fixed raises Gemma T1 from 0.707 to 0.774 and Llama by 0.033, while lowering Aya by 0.0119; these results form a scoring-format sensitivity analysis. Pixtral uses its recorded tokenizer configuration. The models share edited assets but have model-specific admission and labels.

The paper also evaluates single-layer GQA/COCO-derived pools (7B/3B AUROC 0.774 / 0.745) and GQA-derived MCQs (0.838 / 0.717), with model-proposed regions on those two models. Section 4.1 now identifies these sources; Table 2C and Appendix D.3–D.4 report the geometry comparisons.

For a second editing workflow, FLUX.1-Fill-dev uses target-option text directly as its prompt. Seven-model T2 AUROC is 0.6777–0.7789; all paired differences from SDXL include zero. Prediction thus extends to this workflow, where both the editor and prompt strategy change. Section 4.7 and Table 3 summarize this comparison, with full paired results in Appendix I.4 (Table 52).

C6. Predicted regions used for both editing and scoring

The original ablation changed the scoring region while retaining reference-box edits. We now test the workflow in which the predicted box determines both the edit and the score. Qwen3-VL-32B receives the image and question without answers or options; unusable boxes are excluded without human fallback. The editor is FLUX, evaluated on five models.

Predicted-region T2Result
Usable proposals2,081/2,234 (93.2%)
Excluded proposals153 (6.8%)
Predicted boxes, pooled AUROC0.7143–0.8096
Predicted boxes, IoU ≥ 0.50.6635–0.7504
Reference boxes, same high-IoU items0.6545–0.7357
Predicted boxes, IoU < 0.50.7288–0.8320

The score supports response ranking for edits at proposed regions. Low-overlap items contribute to the higher pooled result; high-IoU paired differences from reference boxes include zero. Localization therefore remains relevant to which intervention is being audited.

GLANCE also exceeds area-only prediction by +0.1175 to +0.2109 AUROC across the five models, with all paired intervals positive. A second Qwen3.8-27B locator yields 361 usable boxes from 400 items and T2 AUROC 0.7356–0.8017, again with paired reference-box differences including zero; −g_mask is higher on 3B. These results provide a practical path to proposed-region auditing, with localization yield and stratified results reported explicitly. Section 4.7 and Table 3 summarize these results; Appendix I.5 reports the two locators and overlap strata (Tables 53–55).

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer cweS,

You asked whether the prediction simply reflects the fact that masking and editing affect the same pixels. We compared pairs of models that saw the same edit but differed in whether they changed their answer. The masked-view score ranked the model that changed its answer higher in about 70% of those pairs (Table 25). As the discussion draws to a close, we’d welcome your thoughts on whether this comparison addresses your concern.

Reviewer_6QouOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether you can predict, using only the original image, how a VLM will respond when the answer-relevant region is later edited: will it switch to the new target answer (“follow”) or keep its original answer (“preserve”)? The proposed score, GLANCE, uses the model’s support for its original answer on three views of the unedited image: the full image, a crop of the region, and the image with the region grayed out. The score is the average of the full and crop support minus the masked support. Edits are generated with SDXL and are used only for labels, never for scoring.

On six Visual7W cohorts (Qwen2.5-VL 3B/7B, LLaVA-OneVision, Qwen3-VL-32B, Gemma-3, Qwen3.6-27B), GLANCE reaches AUROC 0.71-0.83 and beats confidence, attention, and gradient baselines, and beats region geometry in five of six cohorts. Probe analyses show most of the predictive signal comes from the masked view. The score does not predict which new answer the model picks (target vs. a third option).

Critical strengths
  1. Clean separation of scoring and labels. The score never sees the edited image, the formula was chosen on separate data and then frozen, and the authors compare against a broad set of baselines, including geometry-matched analyses. This makes the main ranking result credible.

  2. Honest reporting of limits. The paper clearly shows that the score cannot predict the target answer, does not help as an answer selector or training reward, and fails when the region is wrong. This helps readers understand exactly what the method can and cannot do.

Critical weaknesses
  1. The main result is largely expected, and the method adds little over existing measures. If masking a region removes the model’s support for its answer, it is natural that editing that region changes the answer. GLANCE is essentially ERASER-style comprehensiveness applied to images, and it is statistically tied with crop−mask in all six cohorts. The specific formula was picked from six candidates on about 60 items per model with only 12–13 preserve cases. So the contribution comes down to “masked-region support predicts edit sensitivity,” which is a modest step beyond prior evidence-removal work.

  2. The labels mix model behavior with edit quality. A “preserve” label can mean the model ignored the edit, or that the edit did not actually change the answer. In the human audit, 17 of 20 invalid edits were preserve cases, and GLANCE barely separates valid from invalid preserves (AUROC about 0.54). Edits also have known problems: about 13% have black-bar artifacts, about 23% of placebos overlap the region, and 13–15% of labels flip when the edit seed changes. Human validation covers only two models and about 70 items each. As a result, part of the AUROC may reflect which edits succeed rather than how the model uses evidence. The main cohorts are also filtered using post-edit responses, which makes the setting less clean than the “pre-edit prediction” framing suggests.

Questions for authors
  1. What does GLANCE capture that full−mask or crop−mask does not? If nothing, should the paper present the masked-support finding as the contribution rather than GLANCE itself?

  2. How much of the preserve class in the main cohorts comes from failed edits? Can you report T1 AUROC on human-validated edits for more models and larger samples?

  3. Does the result hold with edits from a different generator, or with edit prompts not written by the audited model?

  4. In a realistic setting without ground-truth boxes, how useful is the score? What would a practitioner actually use it for?

  5. Does the effect hold for open-ended answers, or only for four-option multiple choice with letter scoring?

4 comments
AuthorsOpenReview ↗

Response to 6Qou (1/3): Masked-Support Finding and the GLANCE Formulation

Thank you for the framing recommendation and the specific requests on validity and generality. We address Q1–Q5 directly.

R1. Contribution and the role of GLANCE

We adopt your recommendation to foreground the behavioral prediction protocol and continuous masked-support finding. GLANCE combines evidence-removal contrasts: it is the average of full−mask and crop−mask, or Comp − 0.5 × Suff. The empirical contribution is testing these measurements against responses to separately generated edits, while keeping scoring entirely on the original image.

The controls distinguish informative measurements of the same region. Compared with binary pixel-FPVG, continuous GLANCE improves AUROC by 0.1395–0.1719 in seven of eight pools; Gemma's +0.0295 interval includes zero. Adding masked support to matched full/crop probes improves AUROC by 0.1243–0.2426 across eight cohorts, with positive refitting-sensitivity intervals. The increment remains positive after geometry and blank-image controls. The panel contains six Visual7W and two mixed-source cohorts. The fixed-formula comparisons evaluate GLANCE's combined score; these out-of-fold probes quantify the predictive information added by the masked view. Geometry/blank intervals condition on fixed out-of-fold predictions.

An exploratory comparison of discordant model pairs on identical edits provides a further test: GLANCE concordance is 0.6278 [0.6003, 0.6544], and −g_mask is 0.6993 [0.6742, 0.7248], over 2,724 pairs on 627 items. The single-view readout exceeds reverse original-image support by +0.0762 [+0.0442, +0.1087] under a common reference distribution, while GLANCE's difference is +0.0084 [−0.0314, +0.0479]. This supports model-specific prediction by masked support beyond that confidence control.

The formula was fixed on development sets of 60/58 items with 13/12 preserves. The initial seven-model T1/T2 comparison (14 cells, evaluated before the Pixtral/Aya extension) includes −g_mask as a post-hoc single-view comparator:

Fixed-formula comparisonResult
Original six T1 cohorts: GLANCE minus crop−mask−0.011 to +0.010; all intervals include zero
New-source 3B T2: same comparison+0.017 [+0.001, +0.032]
Seven models × T1/T2: maximum gap from each cell's best readoutGLANCE 0.0251; crop−mask 0.0349; full−mask 0.0418; −g_mask 0.0623; paired gap differences favor GLANCE
Expanded 24-cell sensitivity panelIntervals for GLANCE's maximum-gap differences from full−mask and −g_mask include zero

On nine-model T2, −g_mask is significantly higher on five models. These comparisons identify complementary uses: a strong, lower-cost single-view readout for T2 answer-change prediction, and a balanced fixed combination for evaluation across T1 and T2. The revised Introduction and Sections 3.2 and 4.3–4.4 foreground the finding and explain this tradeoff (Figures 3–4; Appendix E.5–E.6, Tables 25–26).

AuthorsOpenReview ↗

Response to 6Qou (2/3): Human-Validated Edits and a Second Editing Workflow

R2. Admission and human-validated edit responses

We address selection and edit validity with separate tests. Removing edited-crop and placebo admission gives nine-model T2 AUROC 0.6928–0.7850 among originally correct items. This predicts any answer change; strict human validity is assessed independently below.

Original sensitivity checkResult
Exclude black-bar-affected edits: 215/1,610 shared assetsT1 AUROC 0.6926–0.8301 across the original six cohorts
Exclude overlapping placebos: 22.24%–25.55% by cohortT1 AUROC 0.7004–0.8345
Two edit seeds, 7B/3BLabels flip on 12.9% / 15.1%; paired T1 AUROC changes +0.017 / −0.048, both intervals including zero

These checks retain predictive discrimination after removing specified defects and show that individual labels depend on the realized edit. They are complemented by the expanded audit of 200 shared edits, rated by three people across five cohorts. Strict validity requires a majority to jointly judge target true, original answer false, and necessary information visible.

The frame excludes earlier rated edits and oversamples assets preserved by any cohort. AUROC uses inverse-inclusion weights; preserve proportions estimate that frame. Failure of strict validity includes ambiguous or non-majority judgments. Raters were blind to model answers, scores and labels; one was an author aware of the hypothesis. Before analysis, 134 author entries affected by a collection-file duplication were excluded and rerated using the frozen interface; the author reported not viewing others' ratings. Strict-validity kappa / pairwise agreement / PABAK are 0.6328 / 0.8200 / 0.6400.

CohortPreserves failing strict validityValid follow / preserveWeighted valid T1 AUROC [95% CI]
Qwen2.5-VL-7B22/48 (45.83%)56 / 260.7459 [0.6122, 0.8614]
LLaVA-OneVision21/43 (48.84%)53 / 220.7599 [0.5945, 0.8953]
Qwen3-VL-32B33/52 (63.46%)65 / 190.7057 [0.5382, 0.8547]
Gemma-3-12B30/63 (47.62%)33 / 330.7177 [0.5842, 0.8434]
Qwen3.6-27B29/47 (61.70%)52 / 180.7483 [0.6125, 0.8690]

The unweighted 32B sensitivity interval includes 0.5. Within preserves, valid-versus-invalid AUROC is 0.4740–0.5441. Human edit validity and observed answer preservation are reported separately.

Under the primary weighted analysis, strict-valid T1 AUROC is 0.7057–0.7599, with every cohort's interval above 0.5. Strict-valid-minus-all-sampled differences are −0.0327 to +0.0087, with all intervals including zero. Prediction therefore persists when the analysis is restricted to human-validated edits: GLANCE distinguishes follow from preserve among successful interventions. Section 4.5 and Table 2B report these results together; Appendix C.3 and G.2 detail the audit (Tables 37–38), and Appendices C.5–C.6 and I.2 retain the original defect and seed checks.

R3. A second editor and direct target-text prompts

Yes: answer-change prediction extends to FLUX.1-Fill-dev with the target option's text used directly as the prompt, removing the model-written prompt step. The comparison now covers seven models.

ModelFLUX T2 AUROCPaired FLUX minus SDXL [95% CI]
Qwen2.5-VL-7B0.7789+0.0159 [−0.0061, +0.0390]
Qwen2.5-VL-3B0.7501+0.0017 [−0.0220, +0.0260]
LLaVA-OneVision0.7630−0.0143 [−0.0386, +0.0104]
Gemma-3-12B0.6777−0.0151 [−0.0416, +0.0123]
Llama-3.2-11B-Vision0.7186−0.0117 [−0.0368, +0.0143]
Pixtral-12B0.7642−0.0171 [−0.0424, +0.0098]
Aya-Vision-8B0.7680−0.0111 [−0.0363, +0.0135]

The original SDXL evaluation already tests prompts from another model: LLaVA-OneVision, 32B, Gemma and 27B use edits prompted by Qwen2.5-VL-7B. FLUX changes both the generator and prompt source, using direct target-option text and T2 answer-change labels without separate human validity ratings. All seven models retain predictive discrimination; paired AUROC-difference intervals between the two workflows include zero. This extends the finding to a second editing workflow with direct text prompts. Section 4.7 and Table 3 summarize this workflow; Appendix I.4 gives the paired results (Table 52).

AuthorsOpenReview ↗

Response to 6Qou (3/3): Practical Auditing and Free Short Answers

R4. Predicted regions and a practitioner workflow

We now use predicted regions for both editing and scoring. A Qwen3-VL-32B locator receives the image and question without answers/options. Of 2,234 proposals, 2,081 are usable; the remaining 6.8% are excluded without human fallback. FLUX generates the edits, evaluated on five models.

Five-model T2AUROC range
Predicted regions, pooled0.7143–0.8096
Predicted regions, IoU ≥ 0.50.6635–0.7504
Reference regions, same high-IoU items0.6545–0.7357
Predicted regions, IoU < 0.50.7288–0.8320

This supports auditing edits at proposed regions. Low-overlap items contribute to the higher pooled result, while paired high-IoU differences include zero. GLANCE exceeds area-only prediction by +0.1175 to +0.2109 across the five models, with positive paired intervals. A second Qwen3.8-27B locator gives 361 usable boxes on 400 items and T2 AUROC 0.7356–0.8017, with reference-box differences again including zero. With this locator, −g_mask is higher on 3B.

A practitioner can use the score to stratify an audit before generating edits and collecting labels. On 500 new-source items (originally correct frames N=449/455; n=100 per design), scores and sampling lists were frozen before editing. Evaluated against the completed pool's reference, the frozen design reduces matched-precision edit/label requirements by 13.1% / 12.6% for 7B/3B with GLANCE and 18.9% / 15.8% with −g_mask. Four of six initial intervals covered the full-pool value.

Together, the proposed-region and prospective-sampling experiments address region localization and budget allocation. Combining measured machine costs with timing from the human audit, the three-rater cost model gives single-view break-even thresholds of 2.0 / 3.1 s per rater-item, below the measured median 10.01 s; GLANCE needs 15.3 / 16.7 s. A fixed 7B reference scorer also reduces matched-precision label requirements for 32B/27B by 12.40% / 12.06% with GLANCE, with lists frozen before target responses on existing edits. These experiments specify how the measurement can allocate an audit budget, beyond its original ranking evaluation. Sections 4.7–4.8 and Tables 3–4 present these applications, with full results in Appendices I.5 and J.4.

R5. Free short answers and full-option-text scoring

Yes: the masked-support finding extends to short answers generated without options. We score each original answer string on the full, crop and masked views using exponentiated mean token log-probability, then compute GLANCE-open = (s_full + s_crop)/2 − s_mask. This removes four-option normalization; the same template-ending tokens are included in each view's score.

Original correctness and semantic change are judged by a text-only Gemma model without human verification. Normalized-string sensitivity replaces the post-edit change rule with normalized-string mismatch on the same original-answer pool.

ModelItemsGLANCE-open T2 [95% CI]−s_maskNormalized-string-label sensitivity [95% CI]
Qwen2.5-VL-7B1,3660.7421 [0.7147, 0.7681]0.78090.7293 [0.7011, 0.7562]
Llama-3.2-11B-Vision1,2740.7000 [0.6692, 0.7292]0.72350.6807 [0.6478, 0.7130]

Both models retain above-chance discrimination under semantic and normalized-string change labels; −s_mask significantly exceeds GLANCE-open on both. These results demonstrate predictive discrimination on short answers generated without options.

Separately, scoring full option texts with aligned prompt/candidate formatting gives T1 AUROC 0.779 / 0.712 on 7B/3B. These are below letter scoring by 0.033 / 0.087, with paired intervals excluding zero. Section 4.7 and Table 3 summarize the free-answer results; Appendices B.2 and I.6 (Table 56) present the two scoring tasks separately.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer 6Qou,

Following your suggestion, we revised the Introduction to foreground the finding that support for the original answer on the masked image predicts the model’s response to a later edit. On your question about edit quality, Table 2B reports both how often edits failed the strict validity check when the model kept its answer and how well GLANCE predicted behavior on edits that passed. Does the revised framing and analysis address your concern, or is there a point you’d like us to clarify?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Response: Main Revisions and New Evidence

Thank you for the detailed reviews. We have added experiments addressing the contribution of the measurement, edit validity, generality, and practical use. The findings support a common conclusion: continuous masked-region support predicts a VLM's response to a subsequent generative edit. GLANCE combines this measurement with full-image and crop support in a formula fixed before evaluation.

Revision guide. In the revised PDF, blue text marks major additions and substantive revisions relative to the original submission; blue captions identify added or substantially revised tables. All section, figure and table references below refer to the revised manuscript.

All intervals below are 95%; supplementary comparisons are unadjusted for multiplicity.

QuestionNew evidence
Does prediction survive removal of post-edit admission?Across nine models, T2 AUROC is 0.6928–0.7850 among originally correct items, without edited-crop or placebo admission. T2 counts any answer change, including third options.
Is there information beyond the shared edited pixels?On identical edits, masked support predicts which of two disagreeing models changes its answer: concordance 0.6993 [0.6742, 0.7248]. Under a common reference distribution, its paired advantage over reverse original-image support is +0.0762 [+0.0442, +0.1087].
Does prediction remain within valid edits?In a three-rater audit of 200 shared edits across five cohorts, weighted T1 AUROC on strictly valid edits is 0.7057–0.7599. Edits failing strict validity account for 45.83%–63.46% of preserves in the audit frame; both results are reported below.
Does it extend beyond the original setup?Added Llama, Pixtral and Aya models; seven-model FLUX T2; five-model predicted-region T2; and free short answers on two models all retain predictive discrimination.
What can a practitioner use it for?Sampling lists frozen before editing demonstrate stratified auditing on 500 new-source items. Completing the pool gives 13.1% / 12.6% fewer required edits and labels at matched precision for GLANCE. Single-view scoring lowers the human-verification cost needed to break even.

T1 retains its conditional follow/preserve definition, and T2 its answer-change definition. These additions complement the negative target-identity, answer-selection and training-reward results. The revised Introduction and Section 3.2 foreground the behavioral prediction protocol and continuous masked-support finding. Sections 4.2–4.8 and Tables 2–4 present the simpler readouts, T2, human audit, generality and audit application.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.