acceptodds
Under review as a conference paper at ICLR 2027

Your Large Vision-Language Model Already Knows How to Localize Changes

Abstract

Visual change understanding necessitates both seeing and saying. However, existing approaches typically realize only one of these capabilities: LVLMs excel at saying changes but struggle to precisely see them, whereas localization-only methods can accurately see changes but lack the semantic capacity to say what those changes represent. To fill this gap, we endow LVLMs with the ability to reliably see changes. By investigating the attention head representations in frozen LVLMs, we find that a small subset of attention heads ( of all attention heads) already encodes change-informative spatial evidence. We excavate these heads by measuring their attention sum deviation from an unchanged self-reference baseline and their spatial consistency across both image directions. Despite being selected without pixel-level supervision, these heads produce responses that closely align with the ground truth. Furthermore, we diffuse their attention through semantic affinities extracted from the frozen vision encoder—rendering stable maps for effective SAM refinements. Extensive experiments across diverse change localization benchmarks demonstrate that our training-free framework substantially surpasses LVLM-based baselines and matches dedicated localization-only approaches. Our results suggest that LVLMs encode an innate spatial understanding of changes via distinct attention patterns, which, combined with their world knowledge and strong descriptive capabilities, enables comprehensive and generalizable change understanding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
change localization, change detection

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_jreqOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether frozen Large Vision-Language Models (LVLMs) already encode spatial evidence for visual change localization, despite their limited ability to explicitly localize changes through textual coordinate generation. The authors propose a training-free framework that identifies change-informative attention heads using two criteria: (1) Attention Sum Deviation, which measures deviations from a semantically unchanged self-reference baseline, and (2) Attention Spatial Consistency, which measures the consistency of spatial responses under image-order reversal.

The selected attention heads are used to extract bidirectional deviation maps, which are aggregated and refined through Semantic-guided Attention Diffusion (SAD) using frozen vision-encoder features. The resulting change response maps are further refined into segmentation masks using SAM or MedSAM proposals.

The authors evaluate the framework across ten LVLMs and diverse benchmarks, including ChangeVPR, MVTecAD, and Brain AD. Experimental results show substantial improvements over existing LVLM-based localization baselines and competitive performance against dedicated localization-only methods, without additional model fine-tuning.

Critical strengths

S1. A compelling training-free approach to change localization using frozen LVLMs

  • The paper presents an interesting approach to extracting change-localization signals from the internal attention representations of frozen LVLMs. Rather than relying on additional grounding supervision or fine-tuning, the authors identify a small subset of attention heads that respond to visual changes and exhibit spatial consistency under image-order reversal. This extends attention-based visual grounding to the more challenging setting of relational change understanding, where the localization targets must be inferred by comparing two images rather than being explicitly specified. The ability to expose such spatial evidence without modifying the underlying LVLM is a meaningful contribution toward integrating change description and localization within a unified framework.

S2. Extensive empirical validation across LVLMs and diverse domains

  • The paper provides strong empirical evidence for the practical effectiveness of the proposed framework. The evaluation covers ten LVLMs with different architectures and parameter scales, as well as substantially different visual domains, including natural scene changes, industrial anomalies, and brain MRI. The method consistently improves upon the corresponding LVLM-based localization baselines and achieves competitive performance against dedicated localization-only methods in several settings. Furthermore, the ablation studies examine the contributions of the head selection criteria, selection strategies, SAD, SAM refinement, and context-aware prompting. These experiments provide useful evidence that the proposed framework is broadly applicable rather than being tailored to a single model or benchmark.
Critical weaknesses

W1. Insufficient justification of Attention Sum Deviation as a criterion for identifying change-responsive heads

  • Although the proposed head selection criteria demonstrate promising empirical results, the motivation behind Criterion 1 requires further justification. Specifically, Attention Sum Deviation measures the signed difference in total attention mass assigned to an image. For example, attention maps [0.30, 0.10] and [0.10, 0.30] yield a deviation sum of zero despite attending to different spatial locations. Thus, potentially useful heads may be discarded before Criterion 2 can assess their spatial consistency.

  • Furthermore, due to softmax normalization over both images and text tokens, attention mass can change without a corresponding change in the target image's attention logits. For example, decreasing the attention logits of the second image alone can increase the attention mass assigned to the first image. Hence, a large deviation does not necessarily indicate sensitivity to semantic changes.

  • The role of the self-augmented baseline also remains insufficiently explained. In particular, Gaussian perturbations do not necessarily represent real-world pseudo-changes such as illumination, seasonal, or viewpoint variations. While the reported ablations support the empirical effectiveness of the proposed criterion, they do not fully establish whether it specifically suppresses pseudo-changes or captures more general context-dependent attention shifts. This distinction matters because identifying inherently change-responsive heads is central to the paper's contribution.

W2. Limited analysis of cross-domain head selection and dependence on task-specific prompts

  • Although the proposed method demonstrates promising cross-domain performance, the generalizability of the selected attention heads remains insufficiently examined. The heads are selected on VL-CMU-CD and fixed across all evaluation domains. Although Appendix C.2 compares head selection on VL-CMU-CD and PSCD, both are natural-scene datasets. This does not establish whether head selection on industrial or medical image pairs would yield similarly informative heads or improve performance in those domains.

  • The anomaly detection prompts in Appendix B.3 explicitly specify category-specific defect types that closely match those in the evaluation benchmarks. Given the substantial performance improvements from context-aware prompting in Table 6, it is difficult to disentangle the contribution of transferable attention heads from that of highly tailored prompts.

Questions for authors

Q1. Justification of Attention Sum Deviation (Related to W1; Sections 4.1-4.2)

  • Could the authors further justify using the signed sum of attention deviations for the initial head filtering? As discussed in W1, this measure can overlook substantial spatial redistribution when positive and negative residuals cancel out. Since Criterion 2 is applied only to heads retained by Criterion 1, potentially informative heads excluded at the first stage cannot be recovered.

  • Have the authors considered alternative measures, such as the L1 or L2 norm of the element-wise deviation map, for the initial candidate selection? Comparing these alternatives in terms of head selection, GT alignment, and final localization performance would help clarify whether signed attention-mass deviation is the most appropriate criterion.

  • As a minor presentation clarification, could the authors also specify how the Head Index in Figure 2 maps to layer and head indices, particularly whether heads are flattened across layers?

Q2. Effect of baseline subtraction and photometric perturbations (Related to W1; Section 4.1 and Appendix D.1)

  • The proposed method uses Gaussian noise to construct the unchanged self-reference baseline. However, Gaussian noise does not necessarily reflect real-world pseudo-changes, such as illumination, seasonal, or viewpoint variations. Could the authors clarify why Gaussian noise was chosen and investigate whether other perturbation types affect head selection and localization performance?

  • Table 12 shows only a modest improvement from Gaussian perturbations (60.9 to 61.8 in average F1 for InternVL3-14B). Could the authors further disentangle the effects of context-aware prompting, unchanged self-reference subtraction, and photometric perturbation? For instance, a controlled comparison varying prompt specificity and baseline subtraction, together with different baseline perturbations where applicable, would be informative.

  • Additionally, evaluating image pairs containing only pseudo-changes, without semantic changes, would help determine whether the proposed baseline actually suppresses nuisance-induced responses rather than general context-dependent attention variations.

Q3. Generality of head selection across domains (Related to W2; Section 4.1 and Appendix C.2)

  • The authors motivate selecting heads on VL-CMU-CD by noting that changed instances appear in only one image per pair, thereby avoiding potential confounding effects when changes occur in both images. Could the authors elaborate on the specific difficulties that arise in the latter setting and explain how they affect the proposed head selection criteria?

  • Although Appendix C.2 compares head selection using VL-CMU-CD and PSCD, both datasets involve natural scene changes. Could the authors investigate head selection using substantially different domains, such as industrial or medical imagery, and compare the resulting heads and localization performance against VL-CMU-CD-based selection?

  • Such experiments could use independent source datasets containing suitable change pairs, without relying on target test annotations, to preserve the validity of the evaluation. This would help clarify whether the selected heads are domain-invariant or whether domain-specific selection provides additional benefits.

Q4. Dependence on category-specific anomaly prompts (Related to W2; Appendix B.3 and Table 6)

  • The prompts for MVTecAD explicitly list category-specific defect types that closely match the benchmark's anomaly taxonomy (e.g., cracks, cuts, holes, and print marks for hazelnuts). Could the authors clarify how these prompts were constructed? Specifically, were they designed a priori based on general domain knowledge, informed by benchmark-specific defect labels or annotations, or refined using validation or test performance?

  • Given the substantial improvement from context-aware prompting in Table 6, could the authors compare three prompting strategies: (i) generic anomaly prompts, (ii) category-aware but defect-agnostic prompts, and (iii) defect-aware prompts explicitly listing anomaly types, while keeping the selected heads and other inference settings fixed? This comparison, ideally including anomaly types not explicitly mentioned in the prompts, would help clarify how much the proposed method relies on benchmark-specific semantic priors.

6 comments
AuthorsOpenReview ↗

Response to Reviewer jreq (1/3)

We deeply appreciate the reviewer’s insightful comments and careful review of our manuscript. We respond to each comment below, grouping related weaknesses and questions. In the revised manuscript, we highlight our major revisions in blue.


[W1&Q1-1] Further Justification for Attention Sum Deviation.

We greatly appreciate the reviewer’s thoughtful comments! To examine whether positive and negative attention deviations cancel out under the signed sum, we investigate heads for which the signed sum is small relative to the absolute deviation (L1 or L2). Specifically, we compute the cancellation ratio (CR) as one minus the ratio of the magnitude of the mean signed sum to the mean absolute deviation, and identify heads with a CR greater than 0.5.

As shown in the table, the reviewer’s concern is reflected in the Criterion 1 candidates: the L1 and L2 norms include 15 and 11 cancellation-affected heads, respectively, whereas the signed sum includes only one. This difference is expected because the signed sum allows positive and negative deviations to cancel, while L1 and L2 retain their magnitudes regardless of sign. However, no cancellation-affected heads remain after Criterion 2. All three variants also yield similar GT alignment scores, select the same top-3 heads, and achieve identical localization performance. Thus, although L1 and L2 initially admit cancellation-affected heads, the spatial consistency requirement in Criterion 2 filters them out, serving as an additional safeguard for robust change localization. We have included this valuable finding in Appendix C.6.

\hline \text{Results} & \text{Signed Sum} & \text{L1 Norm} & \text{L2 Norm} \\\\ \hline \text{Cancellation-affected Heads after Criterion 1} & 1/74 & 15/79 & 11/79 \\\\ \text{Cancellation-affected Heads after Criterion 2} & 0/33 & 0/23 & 0/26 \\\\ \text{GT Alignment} & 0.78 & 0.77 & 0.78 \\\\ \text{Selected Top-3 Heads} & \text{L37H34, L38H38, L38H35} & \text{Same} & \text{Same} \\\\ \text{F1-Score} & 61.8 & \text{Same} & \text{Same} \\\\ \hline \end{array}$$   --- **[W1&Q1-2] Regarding Softmax Normalization.** We are pleased to address this insightful point raised by the reviewer. First, softmax-normalized attention enables comparisons of attention distributions across different forward passes on a common probability scale, providing a concrete view of how much and where an LVLM focuses under a given instruction (L158–L159). Additionally, the table below compares the attention mass assigned to $I_a$ and the pre-softmax logit scores for each input segment. To facilitate comparison, we align the text logit scores across the two forward passes by adding a constant to all logits in one pass, which does not affect the softmax distribution. When $I_b$ replaces $I_a’$, the relative logit scores increase for both image segments, but more so for $I_a$ than for the second image (+2.07 vs. +1.28). This result indicates that, when a change occurs in $I_a$, the logit scores increase for both image segments, but the larger increase for $I_a$ contributes to its higher attention mass; 50% of the logit increase is concentrated within GT regions, despite their accounting for only 6.7% of all pixels in VL-CMU-CD. $$\begin{array}{lccc} \hline \text{Types (L37H34)} & \text{Input } (I_a, I_b) & \text{Input } (I_a, I_a') & \Delta \\\\ \hline \text{Attention mass } (I_a) & 0.16 & 0.03 & +0.13 \\\\ \text{First image logit score } (I_a) & 7.34 & 5.27 & +2.07 \\\\ \text{Second image logit score } (I_b \text{ or } I_a') & 5.74 & 4.46 & +1.28 \\\\ \text{Text logit score} & 9.12 & 9.12 & 0.00 \\\\ \hline \end{array}$$   --- **[W1&Q1-3] Clarification of Head Index** We thank the reviewer for this helpful suggestion. In Figure 2, heads are flattened in layer-major order: the head index is computed as $l \times H + h$, where $l$ is the zero-based layer index, $h$ is the zero-based head index within that layer, and $H$ is the number of heads per layer. We have clarified this in the figure.
AuthorsOpenReview ↗

Response to Reviewer jreq (2/3)

[W1&Q2-1] Effect of Photometric Perturbation.

We select perturbations based on prior studies on robustness and scene change detection [1, 2], which consider 15 types of common corruptions. Following the reviewer’s constructive suggestion, we further evaluate brightness & contrast (illumination), snow (seasonal variation), and crop & resize (viewpoint change). As shown below, head selection remains consistent across all perturbations except snow—yielding the same Top-3 heads—with Gaussian noise achieving the best performance. Overall, the results suggest that augmentation during head filtering helps identify a consistent set of change-informative heads, while its effect during inference is generally modest.

\hline \text{Results} & \text{None} & \text{Gaussian noise} & \text{Brightness \\& contrast} & \text{Snow} & \text{Crop \\& resize} \\\\ \hline \text{Selected Top-3 heads} & \begin{array}{c} \text{L37H34 · L38H38} \\\\ \text{L38H32} \end{array} & \begin{array}{c} \text{L37H34 · L38H38} \\\\ \text{L38H35} \end{array} & \begin{array}{c} \text{L38H38 · L37H34} \\\\ \text{L38H35} \end{array} & \begin{array}{c} \text{L38H32 · L36H21} \\\\ \text{L38H35} \end{array} & \begin{array}{c} \text{L37H34 · L38H38} \\\\ \text{L38H35} \end{array} \\\\ \hline \text{F1-Score (head filtering only)} & 60.9 & 61.6 & 61.6 & 52.2 & 61.6 \\\\ \text{F1-Score (inference only)} & 60.9 & 61.4 & 59.7 & 51.0 & 58.5 \\\\ \text{F1-Score (both)} & 60.9 & 61.8 & 61.0 & 52.1 & 61.7 \\\\ \hline \end{array}$$ &nbsp; [1] Benchmarking Neural Network Robustness to Common Corruptions and Perturbations (ICLR 2019) [2] Differencing-based Self-supervised Pretraining for Scene Change Detection (CoLLAs 2022) --- **[W1&Q2-2] Effects of Prompting, Baseline Subtraction, and Perturbation.** We provide a fine-grained analysis of prompting, baseline subtraction, and perturbations. Following the original protocol, we use InternVL3-14B and select the Top-3 heads using the VL-CMU-CD training set. As shown in the tables below, the context-aware prompt has the largest effect on performance, whereas the choice of baseline perturbation leads to modest variations. These results suggest that baseline perturbation serves as an auxiliary step for refining attention signals passing through change-informative heads, while the semantic definition of changes in the input prompt plays a more essential role. $$\begin{gathered} \begin{array}{lcc} \hline \text{Baseline perturbation} & \text{w/o subtraction} & \text{w/ subtraction} \\\\ \hline \text{None} & 23.5 & 24.5 \\\\ \text{Gaussian noise} & 23.5 & 25.5 \\\\ \text{Brightness \\& contrast} & 23.5 & 24.6 \\\\ \text{Snow} & 23.5 & 25.4 \\\\ \text{Crop \\& resize} & 23.5 & 24.7 \\\\ \hline \end{array} \\\\ \text{<Generic prompt>} \end{gathered}$$ $$\begin{gathered} \begin{array}{lcc} \hline \text{Baseline perturbation} & \text{w/o subtraction} & \text{w/ subtraction} \\\\ \hline \text{None} & 56.6 & 60.8 \\\\ \text{Gaussian noise} & 56.6 & 61.8 \\\\ \text{Brightness \\& contrast} & 56.6 & 61.0 \\\\ \text{Snow} & 56.6 & 59.1 \\\\ \text{Crop \\& resize} & 56.6 & 61.7 \\\\ \hline \end{array} \\\\ \text{<Context-aware prompt>} \end{gathered}$$ &nbsp; --- **[W1&Q2-3] Evaluation on Image Pairs Containing Only Pseudo-Changes.** We construct 300 pseudo-change image pairs from PSCD panoramas, each with an all-zero GT change mask, and evaluate the false positive (FP) rates of the methods. We report two metrics: FP pixels, the average percentage of pixels incorrectly predicted as changed, and FP images, the percentage of image pairs containing at least one predicted change pixel. Although our method yields a higher FP pixel rate than RSCD, it achieves a lower rate than the training-free method GeSCF, while attaining the lowest FP image rate. $$\begin{array}{lcc} \hline \text{Methods} & \text{FP pixels (\\%)} \downarrow & \text{FP images (\\%)} \downarrow \\\\ \hline \text{GeSCF} & 3.85 & 84.4 \\\\ \text{RSCD} & \mathbf{1.86} & 81.5 \\\\ \text{Ours (InternVL3-14B)} & 3.32 & \mathbf{80.1} \\\\ \hline \end{array}$$
AuthorsOpenReview ↗

Response to Reviewer jreq (3/3)

[W2&Q3-1] Confounding Effects When Changes Occur in Both Images.

We thank the reviewer for raising this point. Due to the softmax normalization, concurrent changes in the other image may weaken the attention response to the target image and affect Criterion 1 rankings. For example, as shown below, L38H35 ranks within the top-100 on VL-CMU-CD and proceeds to Criterion 2. When IbI_b also contains a change as in PSCD, its average attention mass on IaI_a decreases from 0.117 to 0.049, and its average attention sum deviation decreases from 0.109 to 0.040. Finally, its Criterion 1 rank drops from 19th to 118th, causing the head to be filtered out before Criterion 2 on PSCD, consistent with Appendix C.2. We generally observe this weakening across heads when changes occur in the other image, but the decline is more pronounced for some heads, including L38H35. Token interactions and dataset biases may contribute to this variation, but understanding why certain heads are more strongly affected remains an interesting direction for future work.

\hline \text{Results (L38H35)} & \text{VL-CMU-CD} & \text{PSCD} \\\\ \hline \text{Avg. attention mass on $I_a$} & 0.117 & 0.049 \\\\ \text{Avg. attention sum deviation} & 0.109 & 0.040 \\\\ \text{Criterion 1 rank} & \text{19th} & \text{118th} \\\\ \hline \end{array}$$ &nbsp; --- **[W2&Q3-2] Generality of Head Selection Across Domains.** Following the reviewer’s insightful suggestion, we additionally used VisA [3] (industrial anomalies) and RESC [4] (retinal anomalies) to filter attention heads in InternVL3-14B. As described in the table below, VL-CMU-CD yields the best performance across all five datasets. Notably, despite their closer domain match, VisA and RESC do not improve performance on MVTecAD and Brain AD, respectively. These results suggest that domain similarity does not necessarily translate into improved change localization performance. One possible explanation is that change-informative heads capture the general concept of *difference* rather than responding only to domain-specific changes. We hypothesize that VL-CMU-CD or PSCD provides a clearer signal for identifying these heads—potentially because its natural images are more familiar to the LVLM. $$\begin{array}{lccccc} \hline \text{Filtering dataset} & \text{SF-XL} & \text{St Lucia} & \text{Nordland} & \text{MVTecAD} & \text{Brain AD} \\\\ \hline \text{VL-CMU-CD} & \mathbf{69.7} & \mathbf{71.7} & \mathbf{48.4} & \mathbf{40.8} & \mathbf{78.3} \\\\ \text{VisA} & 45.7 & 58.5 & 26.1 & 29.1 & 68.0 \\\\ \text{RESC} & 50.6 & 57.5 & 27.9 & 30.9 & 34.1 \\\\ \hline \end{array}$$ &nbsp; [3] SPot-the-Difference Self-Supervised Pre-training for Anomaly Detection and Segmentation (ECCV 2022) [4] BMAD: Benchmarks for Medical Anomaly Detection (CVPRW 2024) --- **[W2&Q4-1] Rationale for Anomaly Prompts.** As described in Appendix B.3, our prompt template is adapted from C3-Bench [5]. We retain its original instructions while simplifying the wording for brevity—maintaining consistency with prior LVLM-based change captioning work without tuning prompts based on validation or test performance. The rationale for specifying context is that anomaly inspection is typically defined by product types or object categories: defects in pharmaceutical pills (e.g., discoloration, contamination, cracks, or faulty imprints) differ substantially from those in metal grids (e.g., bent strands, broken wires, or metal contamination) or leather products (e.g., scratches or color variations). [5] C3-Bench: A Context-aware Change Captioning Benchmark (ECCV 2026) --- **[W2&Q4-2] Effects of Varying Anomaly Prompts.** We thank the reviewer for this insightful suggestion. We evaluate four prompt types with varying levels of anomaly-related information. As shown in the table below, adding general anomaly information substantially improves F1-Score on both datasets (e.g., *Describe the visible abnormality from image 1 to image 2 in one concise sentence*). Category information further improves performance on MVTecAD but has a smaller effect on Brain AD. Lastly, providing specific defect information yields only marginal gains. These results suggest that our framework does not require extensive prior knowledge of the target domain or benchmark—supporting its practical applicability. $$\begin{array}{lcc} \hline \text{Prompt type} & \text{MVTecAD} & \text{Brain AD} \\\\ \hline \text{Fully Generic} & 18.8 \pm 0.7 & 23.8 \pm 2.7 \\\\ \text{Generic Anomaly} & 33.0 \pm 0.3 & 77.0 \pm 0.9 \\\\ \text{Category-Aware} & 39.7 \pm 0.4 & \mathbf{77.6 \pm 1.5} \\\\ \text{Defect-Aware} & \mathbf{40.7 \pm 0.3} & 77.3 \pm 1.8 \\\\ \hline \end{array}$$
Reviewer_77QYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a valuable contribution by demonstrating that frozen Large Vision-Language Models (LVLMs) already possess change detection capabilities and investigating how to extract localization info from the large-scale VLMs. While specialized change detection models have traditionally demonstrated strong performance, LVLMs have gradually improved to achieve competitive results. Rather than developing another task-specific change detection model, this paper approaches the problem from the perspective of how change detection capabilities can be extracted from already trained LVLMs. The experimental results demonstrate that the proposed approach consistently improves performance across multiple LVLM architectures with LLaVA, Qwen, and InternVL, providing evidence of its general effectiveness.

Critical strengths
  • The reviewer finds the key idea of identifying specific layers within LVLMs that respond to change detection and localization, and projecting these responses back into image space to extract change regions, particularly compelling. Furthermore, as shown in Tables 1 and 2, the fact that the resulting performance surpasses that of specialized models suggests that this approach has the potential to establish a new standard for LVLM-based change localization. In this sense, the reviewer believes the study makes a meaningful contribution to the field.

  • Table 1 clearly illustrates the performance gap between specialized change detection models and previous LVLM-based approaches. The authors then demonstrate that applying their proposed localization method to multiple LVLMs successfully extracts strong change localization performance across different architectures. These results suggest that investigating how to extract knowledge and capabilities from LVLMs may lead to alternative yet effective approaches to various downstream tasks. This perspective could open up new research directions beyond conventional task-specific model development with LVLMs and related large-scale models.

Critical weaknesses
  • The reviewer would like to see a discussion of how pre-training on image-language pairs influences the change detection capabilities of pre-trained VL models, or more broadly, their object detection and segmentation capabilities. There are many factors to consider, including the #training-images, the #language-corpus, and the composition of image-text pairs in terms of data and labels, #model-parameters and architectures in terms of models, and supervision strategies and hyperparameters in terms of training. If the paper could examine the differences among the LVLMs evaluated in this study, including their datasets, model architectures, and training procedures, and identify which factors may contribute to improved change detection and localization, the reviewer believes the proposed paper could make an even greater contribution to the field of change detection and LVLms. Since the central claim of this paper is that 'LVLMs already know how to localize changes', the underlying assumption seems to be that sufficiently large models have already acquired certain capabilities for change detection and localization. The reviewer believes discussing this aspect would further enhance the value of the paper.

  • The paper suggests that LVLMs have achieved strong performance in detection and segmentation. However, previous work has reported that SAM struggles with segmentation on industrial datasets (Segment Anything Is Not Always Perfect; https://arxiv.org/abs/2304.05750). This raises the question of whether SAM and other specialized models, which are primarily trained on web- or general-domain data, may have fundamental weaknesses in segmentation-specific models for certain types of images. If these models struggle with segmentation, change detection would naturally become even more challenging. The reviewer believes this possibility should also be considered when discussing the performance of specialized models.

  • What is the relationship among Figures 2, 3, and 4? The reviewer assumes that the paper intends to demonstrate a correlation between the outputs of network layers and spatial localization information. It is that Figure 2 presents the average attention-head responses for each image, Figure 3 illustrates positional bias and the spatial consistency of attention, and Figure 4 shows the relationship between ground-truth locations and attention. However, the reviewer believes the readability of the paper would improve if the authors could explain more clearly how these observations lead to the proposed method. Since these preliminary analyses are important for providing a deeper understanding of the method, it would be helpful to discuss the relationships among these figures, particularly how Figures 2 and 3 relate to Figure 4, either in the main text or in the figure captions.

Questions for authors

Although this is a minor comment, the reviewer wonders whether the 10 evaluated models provide sufficient diversity, given that some of them differ only slightly in parameter counts or training strategies. Table 1 includes 5 LLaVA models, 3 Qwen models, and 2 InternVL3 models. If these models belong to the same versions within their respective model families, their underlying training datasets are likely to be largely similar. Therefore, it is unclear whether these differences are sufficiently meaningful.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 77QY (1/4)

We deeply appreciate the reviewer’s insightful comments and careful review of our manuscript. We respond to each of your comments one by one in what follows. In the revised draft, we mark our major revisions in blue.


[Q1] Regarding Model Diversity.

In the table below, we respectfully elaborate on the diversity of the models used in our empirical evaluation. Based on the information available in the respective papers, the table summarizes the scale and type of interleaved image–text and video training data, where S denotes single-image–text data and M denotes multi-image interleaved text or video data. Unlike the LLaVA-OneVision family, LLaVA-1.5 was not trained on multi-image data, yet our method excavates change-informative knowledge from it—enabling robust performance across diverse benchmarks. The OneVision models also differ in their training data: OneVision-1.5 uses 85M concept-balanced image–text pairs during mid-training, while OneVision-2 incorporates approximately 4.2M spatial reasoning samples during pretraining and 8M re-captioned video samples during SFT. Similarly, Qwen3-VL introduces additional grounding and spatial understanding data compared with earlier Qwen-VL models. These differences in training settings can lead to distinct gradient updates, resulting in different learned representations and attention logits.

Furthermore, differences in model scale also change the number of layers and attention heads, potentially affecting how information is distributed across layers and heads. Nevertheless, Figures 2 and 3 reveal similar change-responsive and spatially consistent attention patterns across scales. Together, these observations suggest that our two criteria can identify change-informative evidence across diverse LVLMs, supporting the generalizability of our method across different model settings—an aspect also appreciated by Reviewers qboo and jreq.

\hline \text{Stages} & \text{LLaVA-1.5} & \text{Qwen2-VL} & \text{Qwen3-VL} & \text{InternVL3} & \text{OV} & \text{OV-1.5} & \text{OV-2} \\\\ \hline \text{Pretraining} & \substack{558\text{K} \\\\ \text{S}} & \substack{- \\\\ \text{S+M}} & \substack{- \\\\ \text{S+M}} & \substack{- \\\\ \text{S+M}} & \substack{558\text{K}+4\text{M} \\\\ \text{S}} & \substack{558\text{K}+85\text{M} \\\\ \text{S}} & \substack{558\text{K}+85\text{M}+4.2\text{M} \\\\ \text{S+M}} \\\\ \hline \text{SFT} & \substack{665\text{K} \\\\ \text{S}} & \substack{- \\\\ \text{S+M}} & \substack{1.2\text{M} \\\\ \text{S+M}} & \substack{21.7\text{M} \\\\ \text{S+M}} & \substack{3.2\text{M}+1.6\text{M} \\\\ \text{S+M}} & \substack{22\text{M}+24\text{M} \\\\ \text{S+M}} & \substack{22\text{M}+24\text{M}+8\text{M} \\\\ \text{S+M}} \\\\ \hline \text{RL} & \substack{-} & \substack{-} & \substack{30\text{K}} & \substack{300\text{K}} & \substack{-} & \substack{-} & \substack{-} \\\\ \hline \end{array}$$ &nbsp; Finally, following the evaluation protocol in Table 1, we further compare our method with Bounding Box + SAM using Idefics3-8B [1] and Pixtral-12B [2]. Our method consistently achieves superior performance on both models. $$\begin{array}{lccccc} \hline \text{Methods} & \text{SF-XL} & \text{St Lucia} & \text{Nordland} & \text{MVTecAD} & \text{Brain AD} \\\\ \hline \text{Idefics3-8B} & 0.0 & 0.0 & 0.0 & 0.6 & 0.0 \\\\ \text{Ours (Idefics3-8B)} & 53.0 & 63.5 & \mathbf{35.9} & 29.5 & 55.3 \\\\ \hdashline \text{Pixtral-12B} & 0.9 & 0.0 & 1.1 & 2.5 & 4.2 \\\\ \text{Ours (Pixtral-12B)} & \mathbf{62.9} & \mathbf{66.8} & 26.1 & \mathbf{33.1} & \mathbf{74.6} \\\\ \hline \end{array}$$ &nbsp; [1] Building and Better Understanding Vision-language Models: Insights and Future Directions (NeurIPS 2024 Workshop) [2] Pixtral 12B (arXiv 2024)
AuthorsOpenReview ↗

Response to Reviewer 77QY (2/4)

[W1] Further Discussion on Origins of Change-Informative Knowledge.

We appreciate the reviewer’s constructive suggestion! In line with our previous comment, understanding how model scale, supervision, and training data shape knowledge in LVLMs is an important research direction for change localization and LVLMs more broadly. As an initial step, Appendix F of the original manuscript analyzes InternVL3-14B checkpoints from different configurations: pretraining, supervised fine-tuning (SFT), and reinforcement learning. We observe that change-informative heads become progressively more distinct across these stages, with SFT contributing most substantially.

To further investigate the reviewer’s suggestion, we conduct three additional experiments using LLaVA-OneVision-7B. We select this model for two reasons: (1) its training recipe and data are publicly available—enabling controlled analysis of the training data (the full training datasets for Qwen-VL and InternVL are not publicly available due to company policies); and (2) its SFT proceeds through successive single-image and multi-image instruction-tuning stages—allowing us to examine how each stage shapes change-informative knowledge at a finer granularity. Specifically, we examine:

  • Model scale: We compare LLaVA-OneVision-7B with its 0.5B and 72B to examine how large a large vision-language model needs to be to inherit change-informative knowledge.
  • Training stages: Following Appendix F, we examine successive checkpoints after pretraining, single-image instruction tuning, and OneVision training (with single-image & multi-image/frame).
  • Training data: We conduct a data ablation from the full training data (approximately 580 GB) during OneVision training by excluding five change-centric datasets: Spot-the-Diff, CLEVR-Change, HQ-Edit-Diff, MagicBrush-Diff, and Birds-to-Words.

Please refer to the next comment :)

AuthorsOpenReview ↗

Response to Reviewer 77QY (3/4)

[Model scale] LLaVA-OneVision-7B consistently outperforms 0.5B across all benchmarks—suggesting that a sufficiently large model may be needed to make change-informative spatial cues accessible through attention heads. However, gains tend to saturate at larger scales for several domains; we observe similar trends in the InternVL and Qwen-VL families.

\hline \text{Scale} & \text{SF-XL} & \text{St Lucia} & \text{Nordland} & \text{MVTecAD} & \text{Brain AD} \\\\ \hline \text{0.5B} & 17.2 & 16.8 & 4.4 & 12.5 & 40.6 \\\\ \text{7B} & 63.5 & 65.9 & \mathbf{43.2} & 35.7 & \mathbf{68.0} \\\\ \text{72B} & \mathbf{67.4} & \mathbf{71.1} & 37.8 & \mathbf{40.4} & 67.5 \\\\ \hline \end{array}$$ &nbsp; **[Training stages]** Interestingly, single-image SFT yields the largest gains in change localization across benchmarks, while subsequent multi-image SFT provides additional improvements. This suggests that instruction tuning plays a major role in making change-informative spatial cues, with multi-image training further enhancing their utility. Meanwhile, as shown in Appendix F, InternVL3 already exhibits reasonable change localization performance after pretraining. One possible explanation is that its pretraining includes interleaved images–text and video–text data (M), providing earlier exposure to multiple images/frames that may contribute to the emergence of change-informative heads. $$\begin{array}{lccccc} \hline \text{Stages} & \text{SF-XL} & \text{St Lucia} & \text{Nordland} & \text{MVTecAD} & \text{Brain AD} \\\\ \hline \text{Single-image Pretraining} & 12.4 & 5.4 & 7.5 & 8.9 & 41.5 \\\\ \text{+ Single-image SFT} & 57.6 & 65.4 & 41.5 & 31.9 & 48.2 \\\\ \text{+ Single-image SFT \\& Multi-image SFT} & \mathbf{63.5} & \mathbf{65.9} & \mathbf{43.2} & \mathbf{35.7} & \mathbf{68.0} \\\\ \hline \end{array}$$ &nbsp; **[Training data]** We successfully reproduced performance comparable to that of the original LLaVA-OneVision-7B model. Notably, removing a comparable amount of random non-change-centric data (42.6K samples) led to a larger drop in localization performance than removing the five change-centric datasets (42.5K samples, comprising 2.7% of the total OneVision training data). These results suggest that change-informative spatial knowledge may not depend solely on change-centric training data; more general multi-image reasoning data may also contribute to its emergence, even without task-specific change supervision.` $$\begin{array}{lccccc} \hline \text{Settings} & \text{SF-XL} & \text{St Lucia} & \text{Nordland} & \text{MVTecAD} & \text{Brain AD} \\\\ \hline \text{Base} & 63.5 & 65.9 & 43.2 & \mathbf{35.7} & 68.0 \\\\ \hdashline \text{Reproduced} & \mathbf{63.9} & \mathbf{66.8} & \mathbf{45.9} & 35.0 & \mathbf{68.8} \\\\ \text{Reproduced w/o 5 change-centric datasets} & 62.2 & 65.6 & 42.8 & 35.2 & 66.5 \\\\ \text{Reproduced w/o 5 non-change-centric datasets} & 60.3 & 64.7 & 41.1 & 34.8 & 67.5 \\\\ \hline \end{array}$$ &nbsp; We have added these analyses to Appendix F of the revised manuscript. We are sincerely grateful to the reviewer for these valuable suggestions, which have helped strengthen our work and clarify its potential for both change detection and LVLM research.
Reviewer_qbooOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the change-informative spatial evidence encoded in the attention mechanisms of Large Vision-Language Models (LVLMs), motivated by their strong change understanding capabilities. Building on these findings, the authors propose a training-free change localization framework. The method first identifies localization-specific attention heads, constructs changed region maps via semantic-guided attention diffusion (SAD), and further refines the maps using SAM. Extensive experiments across diverse LVLMs demonstrate the effectiveness of the proposed framework.

Critical strengths
  1. Insightful and inspiring findings: The paper uncovers the latent, change-informative spatial evidence embedded in LVLM attention maps and effectively leverages it for change localization. This elegantly extends LVLMs from merely saying to both seeing and saying.
  2. Comprehensive experimental validation: The paper conducts extensive experiments across diverse LVLMs and benchmarks, thoroughly demonstrating the effectiveness and strong generalization capability of the proposed framework.
  3. Clear presentation: The paper is well written and easy to follow.
Critical weaknesses
  1. Limitations under viewpoint changes: The deviation maps are computed on each image's own grid, while the final mask is the union of the two maps. When the two views differ, the two maps seem to lie in different spatial regions, which directly influences Criterion 2. Since self-reference image only applies Gaussian noise, attention caused by geometric differences cannot be eliminated.
  2. Coupling with base model quality: The core assumption assumes coupled "saying and seeing" in LVLMsConsequently, the framework's performance likely hinges heavily on the base model’s inherent change-perception capability. The framework may fail or perform poorly when applied to change-insensitive backbones.
Questions for authors

Based on the weaknesses, I have two main questions:

  1. Robustness to viewpoint variations: Could the authors provide a performance breakdown across varying degrees of viewpoint differences on ChangeVPR, or evaluate on benchmarks with realistic viewpoint shifts (e.g., CLEVR-Change [1])?
  2. Analysis on poorly-described samples: In Appendix C.4, only 169 out of 523 samples (caption score ≥5\geq 5) are evaluated. How does the framework perform on the remaining 354 samples? Specifically, (1) If the performance drops noticeably, does it confirm that the framework is heavily limited by the LVLM's native change-captioning ability? (2) If it remains strong, does it challenge the central claim that "seeing" and "saying" are tightly coupled?

Minor Question:

  1. Overstatement in core claim: Is the claim "LVLMs already know how to locate changes" somewhat overstated? Table 5 shows that raw LVLM attention yields nearly trivial performance, with most performance gains driven by SAD and SAM. Does this indicate that the intrinsic localization evidence in LVLMs is too weak to substantiate this claim on its own?

[1] Park, Dong Huk, Trevor Darrell, and Anna Rohrbach. "Robust change captioning." 2019 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2019.

4 comments
AuthorsOpenReview ↗

Response to Reviewer qboo (1/2)

We deeply appreciate the reviewer’s insightful comments and careful review of our manuscript. We respond to each of your comments one by one in what follows. In the revised draft, we mark our major revisions in blue.


[W1-1] Clarification of the Image Grid Used for Both Criteria.

We respectfully clarify that both criteria are computed using deviation maps defined on the same image grid. For example, to analyze the changed instance in IaI_a, we extract attention associated with IaI_a when it appears first in (Ia,Ib)(I_a, I_b) and second in (Ib,Ia)(I_b, I_a). The selected heads are then fixed across all settings during inference. Consequently, our method produces a separate change mask for each image. Then, we take the union of these masks to match the evaluation protocol of benchmarks that require a single change mask. For benchmarks with viewpoint changes (e.g., PSCD-diff in Appendix D.2), evaluation is performed against the GT of the reference image within the shared field of view, following the standard protocol in [1]; our framework inherently supports input-wise change localization.

[1] Robust Scene Change Detection Using Visual Foundation Models and Cross-Attention Mechanisms (ICRA 2025)


[W1-2] Limitations Under Viewpoint Variations.

In Appendix D.2 of the initial manuscript, we evaluate the viewpoint robustness of our method against previous state-of-the-art localization-only methods (GeSCF and RSCD) on the PSCD dataset. Our method achieves F1-Scores of 36.7 (vs 11.1 & 34.1) and 26.9 (vs 5.7 & 23.5) under the Diff-1 and Diff-2 settings, respectively—outperforming baselines that leverage an external alignment module (GeSCF) or an alignment-aware architecture (RSCD). These results suggest that LVLMs can leverage their innate spatial understanding to achieve modest robustness to viewpoint variations. Nevertheless, we acknowledge the reviewer’s concern: the performance decline across methods highlights the ongoing challenge of handling larger viewpoint differences. We have included this limitation in Appendix G and will ensure that it is clearly reflected in the final manuscript.


[W2] Regarding Base Model Quality.

We respectfully clarify that we do not claim a contribution to developing a backbone-agnostic change localization framework. Instead, we aim to identify and exploit change-informative spatial evidence that remains largely hidden in LVLMs’ textual interface—a perspective also appreciated by Reviewers 77QY and jreq. The research focus lies in uncovering latent spatial knowledge in LVLMs, rather than designing a framework independent of the underlying model.


[Q1] Further Experiments on Viewpoint Variations.

We sincerely apologize that CLEVR-Change does not provide pixel-wise change masks for evaluating change localization performance. Instead, as reported below, we evaluate on ChangeVPR under viewpoint perturbations, following the protocol in [1]. The results show a similar trend to those on PSCD, with our method exhibiting modest robustness compared to the baselines. We have added these results alongside the PSCD results in Appendix D.2 of the revised manuscript.

\hline \text{Methods} & \text{Aligned} & \text{ChangeVPR-Diff-1} & \text{ChangeVPR-Diff-2} & \text{Avg.} \\\\ \hline \text{GeSCF} & 48.7 & 40.3 & 26.4 & 38.5 \\\\ \text{RSCD} & 50.7 & 37.7 & 27.1 & 38.5 \\\\ \text{Ours (InternVL3-14B)} & \mathbf{55.3} & \mathbf{44.8} & \mathbf{34.6} & \mathbf{44.9} \\\\ \hline \end{array}$$
AuthorsOpenReview ↗

Response to Reviewer qboo (2/2)

[Q2] Further Analysis of Poorly Described Samples.

We thank the reviewer for this insightful question! We extended the analysis to all 523 samples. Among the 354 samples with poor captions, our method successfully localizes changes (F1 ≥\geq 50.0) in 215 cases (60.7%), compared with 144 of 169 cases (85.2%) among samples with good captions.

These results reveal two key insights: (1) Good captions are often accompanied by successful localization—suggesting that “seeing” and “saying” are related, and (2) Successful localization is also possible for samples with poor captions—indicating that change-informative spatial signals can remain available even when they are not well expressed in text. This suggests that current LVLMs may not fully convey their latent spatial knowledge through their general textual outputs, while our framework can reify this hidden knowledge.

We again thank the reviewer for this thoughtful comment, which has helped us clarify the relationship between “seeing” and “saying.” We have refined our original description of the two as “tightly coupled” to reflect a related but directional relationship. While our caption quality and localization performance are positively correlated (Pearson’s r = 0.38), good captions tend to be accompanied by accurate localization—P(good localization∣good caption)=144/169=85.2P(\text{good localization} \mid \text{good caption}) = 144/169 = 85.2\\%—whereas accurate localization does not necessarily imply good captions—P(good caption∣good localization)=144/359=40.1P(\text{good caption} \mid \text{good localization}) = 144/359 = 40.1\\%. Given the importance of this finding, we have also moved the analysis to Section 5.4 in the revised manuscript.


[Q3] Regarding Our Core Claim.

We acknowledge that raw attention alone yields limited change localization performance. As illustrated in Figure 4(b) in the original manuscript, the selected heads produce sparse responses, so direct binarization naturally captures only a few highly activated points rather than the full extent of changed regions. However, as appreciated by Reviewer 77QY, SAD and SAM project this compact evidence into image space—making it spatially accessible. These components play complementary roles: without informative heads, SAD and SAM lack reliable change cues to form a mask; without SAD and SAM, the selected responses remain concentrated in a few locations. Together, these contributions transform sparse raw attention cues into spatially coherent change masks—synergistically reifying our central claim that LVLMs already know how to localize changes.

Thank the author for the response. I found this overall insight interesting, and the further clarification and experiments have addressed most of my concerns. I have decided to raise my score to 3.

AuthorsOpenReview ↗

Response to Reviewer qboo

Dear Reviewer qboo,

Thank you for your thoughtful feedback and support for our work. We are pleased to hear that our rebuttal has satisfactorily addressed your concerns. Should you have any further questions or suggestions, we would be grateful for the opportunity to address them :)

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Common Response

Dear Reviewers and Area Chairs,

We sincerely appreciate the time and effort you devoted to reviewing our manuscript.

We propose a simple, training-free framework that uncovers and amplifies change-informative knowledge in frozen LVLMs for change localization. We are grateful that the reviewers recognized the clarity of the presentation (qboo), the compelling insights of our analysis (all), the connection between seeing and saying (qboo, jreq), the potential to open a new research direction (77QY), and the strength of the empirical evaluation (all).

In response to your constructive feedback, we have revised the manuscript to include:

  • Further analysis of samples with poor captions (moved from Appendix C.4 to Section 5.4)
  • Effect of domain-specific head filtering dataset (Appendix C.2)
  • Further analysis of attention sum deviation (Appendix C.6)
  • Further analysis of prompt specificity (Appendix C.7)
  • Further ablation of the self-augmented baseline (Appendix D.1)
  • Further ablation of prompting, baseline subtraction, and perturbation (Appendix D.1)
  • Evaluation under varying viewpoints on ChangeVPR (Appendix D.2)
  • Further analysis of origins of change-informative heads (Appendix F)
  • Additional limitations (Appendix G)

These revisions are highlighted in blue for ease of reference. We hope they address your concerns and clarify our contributions to the ICLR community.

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.