acceptodds
Withdrawn from ICLR 2027

InstructCD: A Unified Instruction-Following Framework for Remote Sensing Change Detection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

17 views

Abstract

Remote-sensing change detection has diverged into four paradigms: binary, semantic, referring, and reasoning, with each addressed by specialized models and increasingly driven by natural-language instructions. Existing unified models cover only part of this spectrum. They merge binary and semantic detection, or couple detection with textual question answering, yet none handle all four paradigms in a single model, and none support variable-category referring with rejection or multi-level reasoning. We present InstructCD, an instruction-following framework that recasts every paradigm as one interface: a language query produces a set of <change> tokens, and each token decodes to a mask, so the number and meaning of the outputs follow the instruction. A Multimodal Language Model conditions the queries, a Temporal Scale Adapter encodes pre-change, post-change, and difference evidence, and a Flexible Mask Decoder emits a variable number of masks. To evaluate the harder paradigms, we build directional semantic queries, a category-referring set, and InstructCD-29K, a reasoning benchmark with four difficulty levels whose supervision is recomputed deterministically from ground truth. Under one protocol in which every baseline uses the same or a larger image encoder, InstructCD achieves the best primary metric on all four paradigms, and a single jointly trained model rivals per-task specialists.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Remote Sensing, MLLM, Unified Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mxchOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents InstructCD, a unified framework for four remote sensing change detection tasks: binary, semantic, category referring, and reasoning-based change detection. A Qwen2.5-VL model converts instructions into special token representations used as mask queries, while SAM and FlexMD extract bi-temporal features and decode the masks. The authors also introduce SECOND-Ref and InstructCD-29K and evaluate the framework against task-specific baselines. The results show competitive performance across the four tasks, with a jointly trained checkpoint performing close to the corresponding specialist models.

Critical strengths
  1. I like the idea of handling the four change-detection tasks with the same interface. Binary and semantic change detection, category referring, and reasoning-based requests normally come with different output conventions, so a common set of mask queries is useful. The handling of absent targets also makes sense for an interactive setting.

  2. I think the paper is well executed, and the joint-training result is probably the strongest part of the experiments for me. In Table 4, the single checkpoint gets comparable points versus specific checkpoint. These are teacher-forced results, but they do suggest that the four tasks can share a model without much loss in performance.

Critical weaknesses
  1. My main concern is novelty. The use of special language-model tokens as mask queries is quite close to LISA, PixelLM, and GSVA, and UniChange is relevant to the multi-task aspect. TSA uses the before/after features and their differences, which is a reasonable choice for change detection. However, I could not identify a clear new idea in the language-guided part of the method. It would help if the authors explained more precisely what InstructCD can do that these earlier approaches cannot.

  2. I am also unsure how much the 7B MLLM contributes. The semantic classes and many of the output slots are fixed by the task, and the response format is fairly constrained. Could a smaller text encoder or learned class/query embeddings achieve similar results with the same SAM features and mask decoder? The current ablations tell us something about the decoder, but not much about this choice. Given the instruction-following claim, I think this comparison matters.

Questions for authors
  1. What changes if the MLLM-based query generator is replaced with learned target embeddings or a smaller text encoder, while keeping the same visual backbone, training data, and FlexMD? I think referring and reasoning tasks would be sufficient for a focused comparison. In particular, are there instructions whose interpretation requires reasoning over the two temporal observations and cannot be handled through a predefined land-cover target? I would be more positive if the authors could show a meaningful capability that the simpler alternative fails to capture.

  2. Can the authors report per-target or per-slot results for SECOND-Ref? In particular, how does the model behave when the order of requested categories is changed? These results would help establish whether the output slots follow the requested semantics.

Reviewer_gwLeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents InstructCD, an instruction-conditioned framework that unifies binary, semantic, category-referring, and reasoning change detection through a token-to-mask interface. It also introduces SECOND-Ref and InstructCD-29K by deriving instruction-conditioned targets from existing annotations. The application is relevant, but I have concerns about the diversity and validation of the proposed benchmarks, the evaluation of the claimed multi-target capabilities, and the technical novelty of the architecture.

Critical strengths
  1. The unified interface is practically useful, and the jointly trained model retains most of the performance of the task-specific models.
  2. Computing target masks and structured facts from existing annotations is a sensible way to constrain language generation. Grouping instructions by their source image pair before splitting is also an appropriate safeguard.
  3. The decoder ablation demonstrates a meaningful improvement over the plain-FPN alternative.
Critical weaknesses
  1. My main concern is the reliance on SECOND. Around 83% of the records in InstructCD-29K come from SECOND, which is also used for the semantic and category-referring tasks. Generating more instructions from these images adds useful supervision, but it does not add new scenes, imaging conditions, or semantic categories.
  2. This choice is particularly difficult to understand given the availability of other semantic change detection datasets, such as Hi-UCD and HRSCD. Since the proposed pipeline derives most targets and facts from existing semantic annotations, why was it not applied to another dataset? I do not expect the authors to combine every available dataset. However, showing that the pipeline and model work with another data source and category system would make the contribution more convincing. The xBD component adds disaster-damage examples, but does not answer this question about semantic change detection.
  3. The architecture appears to be an incremental adaptation of existing token-to-mask methods. FlexMD improves over the plain-FPN alternative, which is useful evidence, but a comparison with an existing bidirectional query-based decoder under the same setting would better establish what the proposed design adds.
Questions for authors
  1. I also have concerns about how the new capabilities are evaluated. SECOND-Ref mainly measures the union of the predicted masks. If the model swaps the masks for two requested categories, the union remains unchanged. This means the main metric does not directly verify the claimed correspondence between each target and its output mask. Per-target results, including absent targets within mixed queries, are needed.
  2. For the reasoning data, the paper describes full human inspection, but provides little information about the review process or the errors remaining after inspection. The disaster-cause queries deserve particular attention: an event label can be correct without the cause being identifiable from the two input images. I would like to see evidence that these questions, especially wrong-event negatives, are answerable from the model’s inputs.
  3. I recently came across QAG-360K[1] and was curious how InstructCD would perform on it. There seems to be some overlap between the two settings, particularly in language-guided change grounding. Have the authors considered evaluating their method on this dataset? [1] Show me what and where has changed? question answering and grounding for remote sensing change detection[J]. arXiv preprint arXiv:2410.23828, 2024.
Reviewer_LGmZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents InstructCD, a unified framework for four remote-sensing change-detection settings: binary, semantic, category-referring, and reasoning change detection. An MLLM produces a variable number of change tokens, and each token is decoded into a separate mask. The model also introduces a temporal adapter for combining the two images and a flexible mask decoder for handling multiple requested targets. The authors construct SECOND-Ref and InstructCD-29K for referring and reasoning evaluation. Per-task models achieve the best reported primary metric across the four settings, while a jointly trained model remains close to the corresponding task-specific versions.

Critical strengths
  1. The unified token-to-mask formulation make sense. Representing each requested change with a separate token allows the same architecture to support different tasks and a variable number of outputs. The decoder is also efficient because multiple masks share the same visual features.

  2. The paper includes a broad experimental study. The authors retrain most baselines with a common image encoder, provide module ablations, compare joint and task-specific training, and take care to avoid image leakage between training and test splits. The two new datasets may also be useful to the community.

Critical weaknesses
  1. The referring evaluation does not actually measure whether each output mask corresponds to the correct requested category. The paper merges all predicted masks before computing IoU, so swapping two category masks would not change the score. Since preserving the identity of each requested target is a central motivation for the proposed variable-mask decoder, evaluating only their union leaves this claim insufficiently supported.

  2. The InstructCD-29K demonstrates general reasoning over instructions. Most queries are generated from existing semantic labels, spatial rules, or predefined groups of transitions, and the training and test sets appear to follow the same construction process. The model may therefore learn recurring associations between phrases and label combinations without understanding genuinely new descriptions. Evaluation on held-out templates, compositions, or reasoning concepts would make this claim much more convincing.

  3. Several improvements over the strongest baselines are relatively small, particularly for binary, semantic, and reasoning change detection. Since the paper reports only one training run, it is unclear whether these differences are consistent or within normal training variation. This is especially relevant because the main comparison uses task-specific versions of InstructCD, while the paper’s central contribution is the jointly trained unified model.

Questions for authors
  1. How about the per-category IoU for SECOND-Ref rather than only the union IoU? How often does the correct mask appear in the correct output slot?

  2. How does one joint K-category query compare with running the same model independently on the K categories? This would show whether multi-target decoding provides useful context beyond computational efficiency.

  3. How does the model handle a mixed query containing both present and absent categories? Per-slot acceptance and rejection numbers would be more informative than results on completely negative queries.

  4. Have the authors tested paraphrases, unusual word order, unseen templates, or held-out compositions? A split that holds out language patterns or reasoning concepts would better support the instruction-following claim.

  5. Could the authors include the jointly trained model directly in the main comparison table and report results across multiple training seeds? Some of its margins over the strongest baselines are small enough that run-to-run variation may change the ranking.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.