- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether the responses of commonly used vision-language models (VLMs) to regional image edits can be predicted using measurements obtained solely from the original image. The authors propose GLANCE, a training-free method that can serve as a tool to assess how VLMs are affected by such edits. The method uses three types of inputs to compute its score: the original image, a cropped image of the target region, and an image with the target region masked. GLANCE provides a score that indicates whether a VLM is more likely to follow the edit or preserve its original answer. Experiments show that GLANCE outperforms other observation methods based on attention, geometry, and related features. The authors also evaluate the robustness of the method using alternative masking strategies, editing seeds, and human validity judgments. Overall, this paper investigates a novel research problem. Compared with previous methods, GLANCE makes it possible to predict a model's sensitivity to visual evidence before an edit is performed. However, I still have several concerns.
Critical strengths
-
The paper moves beyond existing post-edit evaluation methods and instead investigates the problem from the perspective of predicting model behavior.
-
The paper evaluates a wide range of mainstream VLMs, and the experimental results are convincing.
-
The paper provides insightful and interesting findings. For example, in Fig.3, masked-region support contributes substantial predictive information beyond visible-image measurements.
-
The discussion is relatively comprehensive and includes the limitations of the proposed approach.
Critical weaknesses
-
The paper explains how the proposed method differs from previous approaches and provides extensive experiments to demonstrate its effectiveness and robustness, but I am still unclear about the specific practical significance or motivation. The paper does not seem to explain this explicitly.
-
The paper only provides edits using two different seeds, but it does not seem to clearly specify or strictly control the extent of the edits. For example, different edits may lead to different model responses. If an image is edited from a giraffe to a moose, the model may follow the edit. However, if the edit changes a moose to an elk, which has very similar visual features, the model may preserve its original answer.
-
The advantage of the proposed GLANCE formulation over simpler alternatives is not established. In Table 1, the results do not clearly demonstrate the advantage of GLANCE over the crop-minus-mask baseline.
Questions for authors
-
Could the authors provide a more clear motivation and some concrete practical examples of why we need GLANCE?
-
Could the authors provide a clearer explanation of the editing process and explain will GLANCE show different performance when dealing with difference levels of editions (for example, moose -> elk and moose -> giraffe)?
-
Could the authors explain why GLANCE is better than the crop-minus-mask baseline?
5 comments
Response to QSjY (2/2): Three-View and Single-View Readouts
Q3. Why use the three-view combination?
GLANCE averages full−mask and crop−mask. Full and crop measure support in different viewing contexts, so combining them has an empirical, endpoint-dependent value. In the original six T1 cohorts, GLANCE and crop−mask have paired AUROC differences of −0.011 to +0.010, with every interval including zero. On new-source 3B T2, a direct paired comparison favors GLANCE by +0.017 [+0.001, +0.032].
The initial seven-model T1/T2 panel (14 cells, evaluated before the Pixtral/Aya extension, with −g_mask as a post-hoc single-view comparator) measures each readout's maximum gap from each cell's best readout:
| Readout | Maximum AUROC gap [95% CI] | GLANCE minus alternative gap [95% CI] |
|---|---|---|
| GLANCE | 0.0251 [0.0189, 0.0419] | — |
| crop−mask | 0.0349 [0.0289, 0.0537] | −0.0099 [−0.0219, −0.0020] |
| full−mask | 0.0418 [0.0343, 0.0801] | −0.0168 [−0.0504, −0.0055] |
| −g_mask | 0.0623 [0.0457, 0.0983] | −0.0372 [−0.0720, −0.0098] |
GLANCE has the smallest maximum gap in this 14-cell panel, with paired intervals favoring it over all three alternatives. In the expanded 24-cell sensitivity panel, the intervals for GLANCE's maximum-gap differences from full−mask and −g_mask include zero. On nine-model T2, −g_mask significantly exceeds GLANCE on 7B, 3B, 32B, Llama and Pixtral.
This measured tradeoff clarifies the use of each readout: the single masked view is a strong, lower-cost choice for T2 answer-change prediction, while GLANCE provides a balanced fixed combination for evaluation across T1 and T2. The revised Introduction, Sections 3.2 and 4.3, and Appendix E.6 (Table 26) explain the predictive finding and these complementary uses.
Follow-up on our response
Dear Reviewer QSjY,
Thank you for asking when a practitioner would use GLANCE. Table 4 gives a concrete example: when estimating how often a model changes its answer after an edit, GLANCE reduced the number of edits and labels needed for the same precision by 13.1% on the 7B model and 12.6% on the 3B model. Does this example address your question about practical u
Response to QSjY (1/2): Practical Use, Cost, and Edit Similarity
Thank you for recognizing the pre-edit prediction problem and the masked-support finding. We address your three questions below.
Q1. Practical use and cost
A concrete use is estimating a model's answer-change rate over a collection of regional edits. A practitioner scores the original items, forms strata, then allocates editing and verification across those strata. Sampling weights recover the population estimate. The benefit is fewer edited and labeled items for the same estimation precision.
On 500 new-source items, the originally correct sampling frames contained 449/455 items for 7B/3B. Each design sampled 100 items per model, with scores and sampling lists frozen before editing.
Completing the remaining items established the full-pool reference for evaluating coverage and comparing the frozen design with simple random sampling at matched precision. Four of six initial intervals covered the full-pool value; subsequent design simulations had approximately 95% coverage. Seven-model retrospective results similarly reduce requirements by 10.6%–19.9%, compared with −0.22% to +1.44% for confidence or visible-only stratification under the same design.
The cost analysis combines measured machine costs with timing from the separate human audit. Machine-only self-scoring costs more than direct labeling; with a 7B reference scorer, three-view scoring adds cost and single-view scoring approximately breaks even when edits must be generated. For human verification, the measured median of 10.01 s per rater-item exceeds the single-view break-even thresholds and falls below the three-view thresholds. This cost model supports single-view scoring for audits requiring human verification.
Using a fixed 7B reference scorer for 32B/27B at n=100 per design reduces matched-precision label requirements by 12.40% / 12.06% with GLANCE and 13.51% / 15.31% with −g_mask. Lists were frozen before target-model responses on existing edited assets; five of six intervals covered the full-pool value. These reductions require no target-model three-view scoring. The revised Introduction and Section 4.8 describe this audit workflow and its break-even conditions (Table 4; Appendix J.4, Tables 59–61).
Q2. Editing process and near versus far targets
The pipeline inpaints region R toward a target answer while retaining pixels outside R. Different seeds vary the realized edit; target swaps vary semantic distance. We therefore tested your moose-to-elk versus moose-to-giraffe concern with both stratification and a paired target-swap experiment.
All six paired AUROC point estimates favor near targets. In the observational T2 strata, the direction varies by model, including −0.055 for LLaVA-OneVision and +0.054 for 3B. Prediction remains informative for close targets, with model-dependent differences across the observational strata.
Near/far is defined by text similarity. A post-result, single-author check of 40 pairs found distinct targets in 38/40 and judged 32/40 near versus 25/40 far targets realizable inside R alone. Near targets were thus more often judged realizable within the original region, providing descriptive context for the paired AUROC results. The editing procedure is detailed in Appendix C.1; Section 4.7 and Appendix I.7 report both analyses, with paired results in Table 57.