- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces M^2Change---a benchmark for identifying, describing, and grounding multiple changes between two images. The authors also propose PDSO, which combines structured fine-tuning, change-focused distillation, and set-level optimization. Applied to Qwen3.5-4B, PDSO outperforms the evaluated MLLM-based baselines across multiple metrics on M^2Change.
Critical strengths
I believe that a significant strength of this paper lies in its initial problem formulation: "A joint evaluation must therefore account for missing or redundant changes while checking whether each description and its regions refer to the same event (Line 053)."
Contemporary change-understanding literature---including localization with pixel-wise masks or bounding boxes, as well as change captioning with natural language descriptions---has largely focused on evaluating performance within each modality independently. As MLLMs become increasingly capable of processing multiple images, jointly evaluating these two capabilities is an important problem. In particular, beyond measuring the quality of individual modality-specific outputs, assessing whether the predicted localization and textual description are reliably linked to the same change event is a valuable and important research direction for trustworthy change understanding.
However, despite its importance and the paper's motivation, I think that the current study does not fully address the formulated problem for the reasons described below.
Critical weaknesses
I outline the weaknesses that, in my point of view, limit the paper’s completeness; they are listed in descending order of importance.
[Limited Literature Scope and Unclear Positioning] The scope and positioning of this paper are currently unclear. The paper addresses visual change understanding through change grounding/localization and change captioning, but its experiments and discussion appear to focus primarily on MLLM-only approaches. This literature scope does not sufficiently address the broader research trends on change detection, change localization, and change captioning, which have been extensively studied before the emergence of MLLMs.
Conventional change detection and localization methods already take image pairs as input and identify spatial changes through pixel-wise masks or bounding boxes [1,2,3]. Additionally, bounding-box predictions can be effortlessly converted into masks using segmentation foundation models such as SAM. Including representative conventional methods as baselines and evaluating them on the proposed dataset would help readers assess the dataset’s compatibility and understand the relative strengths and limitations of proposed approaches in the broader scope of change understanding works. Similarly, conventional change captioning methods [4,5] also process image pairs and generate descriptions of their differences, yet they are not included as baselines.
[Missing Prompt Details and Prompt Sensitivity] As highlighted by C3-Bench---which the paper cites---MLLMs’ change understanding can be sensitive to contextual prompts that specify what should and should not be considered a change. Without such guidance, generic prompts may leave models to rely on their own assumptions about changes, potentially affecting both localization and captioning performance. However, the paper does not report the prompts used in its experiments or examine how different prompt formulations affect the results. This hinders reproducibility and makes it difficult to determine the robustness of the proposed approach and constitutes a substantial gap in the experimental design that cannot be addressed by a reproducibility statement alone.
[Limited Justification of the Proposed MCJS Metric] Although the proposed MCJS metric is presented as a major contribution in the introduction, its detailed explanation is deferred to the appendix. Moreover, the metric appears to combine localization and captioning scores rather than explicitly evaluate the correspondence between a generated description and the localized change. From my perspective, simply aggregating these scores obscures a model’s individual capabilities rather than yielding additional insight. Also, the metric relies on conventional captioning metrics (e.g., BLEU), whose alignment with human judgments of semantic correctness and relevance may be limited. Complementing these metrics with a well-specified LLM-as-judge evaluation, ideally validated against human judgments, would provide a more informative assessment of the methods.
[Missing Dataset Scale Comparison] Finally, Table 1 would benefit from reporting concrete dataset statistics, such as the number of image pairs, annotated changes, and captions, alongside those of existing benchmarks. I believe that the proposed test-only dataset is relatively limited in scale: CLEVR-Change and C3-Bench each provide approximately 4k test image pairs for change captioning, while the PSCD dataset [3] provides 1,155 image pairs with change-mask annotations (more laborious annotation than box), compared with 1,063 pairs in the proposed dataset. To establish its competitiveness despite this smaller scale, the evaluation metrics and experiments should explicitly leverage its distinguishing feature---two simultaneous annotations per image pair. However, as discussed above, MCJS appears to be a simple combination of existing metrics rather than an evaluation of whether the generated captions are grounded in the same event or aligned with human judgments.
[1] The Change You Want to See (WACV 2023)
[2] Towards Generalizable Scene Change Detection (CVPR 2025)
[3] Robust Scene Change Detection Using Visual Foundation Models and Cross-Attention Mechanisms (ICRA 2025)
[4] Distractors-Immune Representation Learning with Cross-modal Contrastive Regularization for Change Captioning (ECCV 2024)
[5] DECIDER: Difference-aware Contrastive Diffusion Model with Adversarial Perturbations for Image Change Captioning (AAAI 2025)
Questions for authors
I currently lean toward a solid rejection in its current form. Yet, if the authors can satisfactorily clarify and address the weaknesses outlined above, I would be happy to reconsider the assessment during the discussion period.
2 comments
Thank you for the thoughtful response. It addressed some of my concerns, particularly those regarding the motivation for constructing new dataset.
Though, I still feel a fundamental gap between what MCJS can measure---a central contribution of the paper---and the paper’s claim that it jointly evaluates captioning and grounding.
In light of your response, I will raise my score to 2 for now and make my final assessment after considering the discussion during the AC--reviewer discussion.
Thank you again for taking the time to address my comments :)
Thank you for authors' detailed response
Thank you for the detailed response. The authors’ explanation has partially clarified my understanding of the intended interpretation and evaluation of MCJS. However, I remain concerned about (1) whether it truly measures the association between a generated description and its predicted bounding boxes "independently" of their correctness with respect to the ground truth; and (2) the limited significance and relevance of the benchmark compared to existing literature.
[Regarding MCJS Metric]
For example, suppose the ground-truth change is a color change in a cup, but a model incorrectly describes a color change in a plate and predicts bounding boxes around that plate in both images. Although both predictions are incorrect with respect to the ground truth, they are internally consistent: the description and bounding boxes refer to the same object and the same purported change. My understanding is that MCJS would assign a score of zero in this case, despite this consistency between the textual and spatial predictions. Consequently, MCJS appears to combine description, label, and grounding scores associated with the same ground-truth change, rather than directly assess whether the generated description and predicted bounding boxes refer to the same change event. While this provides a fine-grained evaluation of joint correctness, it does not necessarily establish that the textual and spatial predictions are reliably linked independently of ground-truth correctness---which makes the claim in Line 053 somewhat overstated.
[Limited Significance and Relevance of the Benchmark Relative to Existing Work]
In Lines 188–189, the authors state, “The changes are then generated together while preserving the viewpoint, illumination, background, and other scene content.” However, distinguishing meaningful scene changes from these pseudo variations is itself a long-standing challenge in change understanding literature. Rather than directly addressing this challenge, the proposed benchmark appears to sidestep it through a simplified problem formulation. It therefore remains unclear how well findings on the proposed benchmark extend to settings where such variations are present (as in real-world scenarios).
Furthermore, the motivation for constructing a new synthetic dataset from LVIS photographs under such controlled conditions remains insufficiently clear. Existing real-world change-understanding datasets already contain the challenging variations described above. Why not strengthen these datasets with change captions or bounding-box annotations? Such an approach could potentially retain the proposed change categories—object, attribute, counting, and relation—while increasing data diversity/scale and capturing more realistic conditions. Overall, the authors have neither sufficiently justified the need to construct a new synthetic dataset from scratch rather than extend existing real-world datasets nor adequately validated the benefits of this design choice.
Finally, as a benchmark and dataset paper at ICLR, I believe the work should provide more insight into the limitations of existing models, beyond aggregate performance and the improvement of the proposed method. While the paper provides qualitative examples (e.g., Figure 4 and Appendix E, Figure 5), it does not report detailed statistics on the prevalence of different error types or explain which changes are particularly challenging for strong baselines. An analysis of errors such as missed changes, incorrect descriptions, and mismatched description–region associations—potentially broken down by change type and number of changes—would help clarify where current models struggle and what limitations remain even after applying the proposed method.
For these reasons, I remain inclined toward rejection at this stage.