acceptodds
Under review as a conference paper at ICLR 2027

Seeing What Changed: Embodied Change Grounding with Selective Cross-View Perception Optimization

Abstract

Embodied agents often revisit the same environments, where objects may appear, disappear, move, or change state between visits. To study how an agent can tell what has changed and where, we formalize Embodied Change Grounding (ECG): given a reference image, a query image captured after the scene has changed, and a language instruction, a model must identify the referred change and segment the changed object, which may lie in either image. As the two egocentric views are unaligned, this requires cross-view correspondence before localization. We introduce ECG-Bench, a benchmark of 10K samples from 1,859 simulated, industrial, and real revisited scenes with six change types, and ECG-50K, a training set with Change-CoT traces. Our evaluation shows that current MLLMs, reasoning segmentation frameworks, and RL post-training methods all fall short on ECG, and that models often produce fluent reasoning that is not grounded in the cross-view evidence. We therefore propose ECG-Grounder, whose CoT-Guided Cross-View Decoder selects the view containing the target, aligns the two views, and decodes the mask under the guidance of the reasoning chain. We further introduce Selective Cross-View Perception Optimization (SCPO) to train the model, rewarding reasoning chains whose outputs are more sensitive to removing the change-critical region than to a matched corruption elsewhere, as measured at high-entropy tokens within GRPO. ECG-Grounder with SCPO achieves the best results on ECG-Bench and also generalizes to other multi-image reasoning benchmarks such as MIG-Bench.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Embodied Change Grounding, Reasoning Segmentation, Cross-View Correspondence, Chain-of-Thought Reasoning, Reinforcement Learning, Embodied AI, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xmbuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Embodied Change Grounding (ECG): given two photos of the same scene and an instruction, segment the object that changed. It presents a benchmark with 10k samples, and a 50k training set. The proposed method selects which view holds the target, aligns the two views, and decodes the mask from its reasoning. The authors further introduce SCPO, an RL reward that favors reasoning that depends on the region that actually changed, rather than on unrelated parts of the image. Experiments show that existing MLLMs and standard RL perform poorly on ECG, while ECG-Grounder with SCPO achieves the best results.

Critical strengths

ECG is a well-motivated problem that goes beyond standard change detection. The benchmark is also reasonably diverse, covering multiple types, scenes, and viewpoint changes.

The cross-view decoder explicitly models correspondence between the two views, and SCPO encourages reasoning that is grounded in the relevant change evidence. The method is further supported by extensive ablations and transfer experiments.

Critical weaknesses

The statistical robustness of the reported improvements is unclear. Although SCPO achieves the best overall results, its margins over the strongest closely matched baseline, VPPO, are relatively modest (+1.7 and +2.1). Some component-level gains are smaller still: the matched nuisance branch contributes +1.2 and entropy-based token selection only +0.5. The paper does not report results across multiple random seeds, repeated-run variance, confidence intervals, or statistical significance tests for these comparisons. Given the stochasticity of RL post-training, it is difficult to determine whether these relatively small margins are robust or partly due to run-to-run variation.

Questions for authors
  1. The paper states that ECG-50K contains no real-world pairs and that the real split of ECG-Bench therefore measures synthetic-to-real transfer. Could the authors report performance separately on simulated, industrial, and real-world subsets?

  2. Could the authors evaluate a frontier MLLM (e.g. GPT, Gemini, etc.) + state-of-the-art segmentation model (e.g. SAM 3) baseline on ECG-Bench? This would provide a stronger reference point for assessing the necessity of the proposed ECG-specific architecture and training.

  3. How does the model behave when multiple changes occur simultaneously, or when no relevant change is present? These cases would test whether the model can handle more realistic change patterns rather than assuming exactly one target change per pair.

  4. How does the trained model perform when given the instruction with only the query image, or only the reference image, or a mismatched reference-query pair? Some instructions may contain strong cues about the answer, for example by naming the change type and location ("the newly appeared object on the lower-right floor"). If single-view and mismatched-pair performance is substantially lower than the full two-view setting, this would provide stronger evidence that ECG-Bench genuinely requires cross-view reasoning rather than allowing the model to exploit linguistic or single-image shortcuts.

6 comments
AuthorsOpenReview ↗

Response to Reviewer xmbu (Part 1/3)

We thank the reviewer for the careful and constructive comments. We ran new experiments for each point and report them below. All new results are in the revised PDF (in blue).


W1. Statistical robustness of the reported gains

We train GRPO (the VisionReasoner† row of Table 2), VPPO, and SCPO three times each with different seeds, on both Qwen2.5-VL-7B and Qwen3.5-9B, under identical settings.

Table A. Results over three random seeds (AVE, %, mean ± std).

MethodQwen2.5-VL-7BQwen3.5-9B
GRPO32.3 ± 0.734.5 ± 0.5
VPPO33.7 ± 0.436.5 ± 0.3
SCPO (Ours)35.5 ± 0.338.7 ± 0.2

SCPO has the highest mean and the lowest standard deviation on both backbones. Its gains over VPPO (+1.8 and +2.2) are several times larger than the standard deviations (0.2–0.4), so they are unlikely to be run-to-run variation.

We also repeat the two component ablations with three seeds (Qwen2.5-VL-7B).

Table A2. Component ablations: AVE (%) over three random seeds, and average CoT length.

VariantTable 43-seed mean ± stdAvg. CoT tokens ↓
SCPO w/o nuisance branch34.434.3 ± 0.241.5
SCPO w/o entropy selection35.135.0 ± 0.149.5
SCPO (full)35.635.5 ± 0.340.6

The seed means stay within 0.1 of the numbers in Table 4. The gain from the nuisance branch (+1.2) is four times the largest standard deviation (0.3), so it is robust. The gain from entropy selection in AVE (+0.5) is consistent across seeds but small. Both tables are now Table A11 (App. E.10) of the revised PDF.

Entropy selection mainly affects the length of the reasoning (last column of Table A2). Without it, the average CoT length rises from 40.6 to 49.5 tokens, about the level of PAPO (47.7) and VPPO (48.8), while removing the nuisance branch leaves it almost unchanged (41.5). We believe the reason is that the reward is measured only at the few tokens where the model makes its decisions, so extra descriptive text earns no reward.


Q1. Results on the simulated, industrial, and real-world subsets

We report each subset separately below (now Table 6 and Table A7 of the revised PDF). The real-world subset (993 samples) is a pure synthetic-to-real test, because ECG-50K has no real-world pairs.

Table B. mIoU (%) on each subset of ECG-Bench (Qwen2.5-VL-7B backbone).

MethodSimulated (7K)Industrial (2K)Real-world (1K)All
Qwen2.5-VL-7B + SAM 2 (training-free)9.49.09.39.3
LISA23.420.418.622.3
ECG-Grounder (CoT-SFT)30.432.425.430.3
+ GRPO32.935.625.232.6
+ PAPO33.637.125.833.5
+ VPPO34.137.425.133.9
+ SCPO (Ours)35.639.328.335.6
  • Our change decoder is above LISA on every subset: +7.0 (simulated), +12.0 (industrial), and +6.8 (real-world).
  • SCPO transfers best to real scenes: 28.3, which is +3.1 over GRPO, +3.2 over VPPO, and +2.5 over PAPO.
  • GRPO and VPPO improve the simulated and industrial subsets but not the real one (25.2 and 25.1 vs. 25.4 for CoT-SFT). SCPO is the only RL method with a clear gain there (+2.9).

Q2. Frontier MLLM + state-of-the-art segmentation model

We evaluate seven proprietary MLLMs combined with SAM 3 (now Table 2 and App. E.4 of the revised PDF).

Table C. Frontier MLLM + SAM 3 on ECG-Bench (mIoU, %).

ModelAddMissingReplaceRotateRelocateStateAVE
GPT-6.1 Terra56.554.373.343.358.169.259.1
Gemini 3.8 Flash51.152.175.640.856.467.357.2
Claude Opus 5.548.949.867.040.544.567.253.0
Claude Haiku 5.538.531.543.727.038.556.639.3
GPT-6 Luna38.527.244.133.338.656.439.7
Claude Sonnet 4.619.920.023.521.926.528.923.5
Claude Opus 4.66.48.613.111.210.117.711.2
ECG-Grounder + SCPO (Qwen3.5-9B, Ours)36.630.339.338.426.561.338.7
  • Flagship models are strong but far from solving the task: GPT-6.1 Terra, Gemini 3.8 Flash, and Claude Opus 5.5 reach 53.0–59.1 AVE, and they are expensive to call. Fig. A14 of the revised PDF visualizes typical failures of the two strongest models, which read the apparent motion caused by the viewpoint change as an object change.
  • Our 9B model (38.7 AVE) is on par with the non-flagship Claude Haiku 5.5 (39.3) and GPT-6 Luna (39.7), and above both on Rotate and State.
  • Our model is still about 20 points below the best pipeline, so small models have much room to improve in cross-view reasoning.

These results support ECG-specific training: the same Qwen3.5-9B reaches only 16.0 AVE without training (Table 2) and 38.7 after our post-training.

AuthorsOpenReview ↗

Response to Reviewer xmbu (Part 2/3)

Q3. Multiple simultaneous changes, and cases with no relevant change

(1) Multiple changes. ECG-Bench already covers this case to a large extent.

Table D. Multi-change statistics of ECG-Bench.

SplitSamples whose pair has ≥2 changesMax. changes in one pairSamples with same-category distractors
Real85%1058%
Synthetic88%696%

Most samples (85% of the real split and 88% of the synthetic split) come from pairs with two or more changes, and the rest have a single change. We also split the real part by the number of changes:

Table E. Results on the real part, split by the number of changes (mIoU, %).

Real partShareGRPOPAPOSCPO (Ours)
Pairs with ≥2 changes85%24.124.727.1
Pairs with a single change15%31.432.435.5
All100%25.225.828.3

Referring to one change among several is clearly harder for every method (24.1–27.1 vs. 31.4–35.5). SCPO is the best in both groups, and on multi-change pairs it is +3.0 over GRPO and +2.4 over PAPO. Table E is now Table A8 (App. E.7) of the revised PDF.

(2) No relevant change. ECG-Bench currently follows the target-present setting: every instruction refers to a change that really exists. This is the common setting in reasoning segmentation and grounding (e.g., LISA, Seg-Zero, VisionReasoner). It tests whether a model can locate and segment the target under a complex instruction, not whether the target exists.

We agree with the reviewer that in real embodied use the described change may not have happened, so this ability is worth testing. We therefore build 2,400 negative instructions at two difficulty levels (1,200 each, about 24% of the benchmark size):

  • N0 (unrelated instruction): the instruction is borrowed from the other domain (warehouse ↔ home), and we make sure that its landmark does not exist in the current scene. It tests whether the model rejects a clearly unrelated instruction.
  • N1 (hard negative): we keep the location and the landmark of the original instruction, but describe a change that did not happen, by flipping the change direction (polarity flip) or swapping the change type (type swap). Every negative is checked against all annotated changes of the pair.

Step 1: no extra training. We first test CoT-SFT, GRPO, PAPO, and SCPO as they are. Even when the prompt explicitly asks the model to output NO_CHANGE if no change matches, all four models still tend to output a mask. The AUROC of their rejection scores is only 0.46–0.59, which is close to chance. So changing the prompt alone does not give reliable rejection, and this is a limit of target-present training.

Step 2: training with negatives. We then add the two types of negative samples to the training set and retrain CoT-SFT, GRPO, VPPO, and SCPO under the same settings. We check (i) how often the model correctly rejects a negative instruction (N0 / N1 Acc.), (ii) how often it wrongly rejects a positive one (False Rej.), and (iii) whether it keeps its grounding accuracy on the original benchmark.

Table F. Results after training with negative instructions.

MethodOriginal mIoU ↑Retrained mIoU ↑ΔN0 Acc. ↑N1 Acc. ↑False Rej. ↓
CoT-SFT30.329.5−0.876.832.46.9%
GRPO32.631.5−1.175.139.85.4%
VPPO33.931.6−2.378.233.48.0%
SCPO (Ours)35.634.3−1.379.540.85.5%
  • With negative samples in training, all methods learn to reject unrelated instructions (N0 Acc. 75.1–79.5%), at a small cost in grounding accuracy (−0.8 to −2.3 mIoU).
  • SCPO gives the best trade-off: the highest rejection accuracy on both N0 and N1, the highest mIoU after retraining (34.3), and a low false rejection rate (5.5%).
  • Hard negatives remain difficult for all methods (N1 Acc. 32.4–40.8%). Deciding whether a specific change really happened needs fine cross-view comparison, and we see this as an open challenge; it is listed as a limitation in App. H.

These results are now in App. E.8 of the revised PDF (Table A9). We also added a visualization (Fig. A19) with four real cases of SCPO: a rejected and a non-rejected instruction for each of N0 and N1, with the reasoning chain of each. We will release the negative set as an extension of ECG-Bench.

AuthorsOpenReview ↗

Response to Reviewer xmbu (Part 3/3)

Q4. Single-view input and mismatched reference–query pairs

We ran all three tests. We keep each trained model and the instruction fixed and change only the visual input:

  • Reference only / Query only: the model sees only one of the two images.
  • Mismatched pair: we keep the image that holds the ground-truth mask and replace the other image with the same-role image (reference for reference, query for query) from a donor sample. The donor comes from the same data source but a different scene, and shares no image file with the original sample. Using the same source keeps the image style and resolution unchanged, so the model cannot notice the swap from style alone.

In the tables below, "Drop in AVE" and the values in brackets are measured against the full-pair result of the same model (relative drop in %). In the revised PDF, Tables G and I are Table A4 (App. E.5) and Table H is Table 5 (Sec. 5.4).

Table G. SCPO under different visual inputs (mIoU, %, Qwen2.5-VL-7B).

InputAddMissingReplaceRotateRelocateStateAVEDrop in AVE
Reference only0.716.30.71.60.73.13.9−31.7 (−89%)
Query only4.50.66.35.84.123.47.5−28.1 (−79%)
Mismatched pair13.114.513.715.69.421.114.6−21.0 (−59%)
Full pair33.827.136.235.623.457.735.6–

Table H. All post-training methods under different visual inputs (AVE, %).

MethodReference onlyQuery onlyMismatched pairFull pair
ECG-Grounder (CoT-SFT)6.1 (−80%)15.5 (−49%)13.3 (−56%)30.3
+ GRPO4.2 (−87%)11.3 (−65%)14.0 (−57%)32.6
+ PAPO4.1 (−88%)8.9 (−73%)12.5 (−63%)33.5
+ VPPO4.5 (−87%)7.1 (−79%)12.8 (−62%)33.9
+ SCPO (Ours)3.9 (−89%)7.5 (−79%)14.6 (−59%)35.6

Table I. Claude Haiku 5.5 + SAM 3 under different visual inputs (mIoU, %).

InputAddMissingReplaceRotateRelocateStateAVEDrop in AVE
Reference only1.711.03.53.60.92.23.8−35.5 (−90%)
Mismatched pair14.413.118.513.418.621.216.5−22.8 (−58%)
Full pair38.531.543.727.038.556.639.3–

What the results show:

  1. Single-view and mismatched-pair results are substantially lower than the full two-view setting, so ECG-Bench needs cross-view reasoning. Every method drops from 30.3–35.6 AVE to 3.9–15.5, a relative drop of 49–89%: 80–89% with the reference only, 49–79% with the query only, and 56–63% with a mismatched pair. The frontier pipeline (Claude Haiku 5.5 + SAM 3) drops in the same way, from 39.3 to 3.8 (−90%), 6.3 (−84%), and 16.5 (−58%).

  2. A few change types keep some signal from one image. With a single image, almost every type falls to a very low level. The exceptions are expected: a missing object is visible in the reference view (Missing, 16.3 with the reference only), and a state change is often visible in the query view (State, 23.4 with the query only). With a mismatched pair, some types also do not fall to near zero, because the image that holds the target sometimes gives hints about it. So some samples can be roughly guessed from the image that holds the target. Even in this setting, every model loses more than half of its AVE (56–63%), and all types stay far below the full-pair results.

  3. Many of the remaining correct cases are lucky guesses. Figs. A15 and A16 (App. E.5) of the revised PDF show mismatched-pair cases where the prediction is still correct. In a part of them, the model hallucinates a change and happens to pick the right object. Locating the target reliably still needs cross-view reasoning.

  4. Our instructions are disambiguated to limit language shortcuts. A precise reference such as "the newly appeared object on the lower-right floor" is used when the pair has several changes, distractors, or objects of the same category, so the text alone does not pick out the target. When the pair has only one change, we give only a coarse description of the change. This agrees with the statistics in Q3: 85% of real samples and 88% of synthetic samples come from pairs with two or more changes, and 58% / 96% contain distractors of the same category. This procedure is now described in App. E.5.

Reviewer_8WuiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Embodied Change Grounding (ECG), a novel problem setting in which physical agents detect and describe changes in real-world environments. In particular, the task involves identifying and recording changes in the same location observed at different times, including objects that appear, disappear, move, or change state. Given the recent progress toward Physical AI, the reviewer believes this problem setting is particularly important and timely. The experimental results also demonstrate clear improvements over training-free approaches, highlighting the effectiveness of the proposed benchmark and additional training.

Critical strengths
  • The paper introduces not only a novel task setting but also a benchmark called ECG-Bench in order to effectively evaluate an ability to detect and describe changes. With approximately 10k sample scenes, the dataset covers a substantial number of diverse scenarios. 6 types of change labels are annotated across simulation, industrial, and real-world environments. As illustrated in Figures 1 and 2, the annotations are highly detailed and complex, making the dataset practical for agents such as robots and autonomous vehicles operating in real-world environments. Figure 2 presents 6 categories of changes, namely missing, relocation, existence, replace, state, and rotation, which reflect realistic situations that may occur in the physical world. Change detection generally requires considerable time and effort when performed manually by humans, whereas this study also explores the potential for automation in physical environments. By providing both change localization and textual descriptions, the benchmark could enable humans to quickly recognize what has changed and take appropriate actions. In this sense, the reviewer believes the paper has proposed a well-motivated and practical benchmark, along with a corresponding method.

  • The reviwer also finds the motivation behind Table 1 particularly compelling. The inclusion of egocentric observations, cross-time comparisons, change detection, and language reasoning, together with the diversity of change types and the coverage of real-world scenes, represents an important strength of the proposed benchmark. The reviewer believes including the dataset sizes in the comparison would make its advantages even more apparent. Nevertheless, Table 1 is already effective in presenting the characteristics of the benchmark and substantially improves the readability of the paper.

Critical weaknesses

Regarding the benchmark evaluation, only Qwen-based methods appear to have been evaluated. Is this a limitation specific to the proposed approach, or is it a broader issue shared by existing methods in this research area? The reviewer would like to understand whether similar approaches can be applied to other MLLMs or whether Qwen is particularly well suited to solving this task. This could become a concern for researchers who wish to enter this research area and develop new models for Physical AI based on the proposed benchmark, ECG-Bench, as it remains unclear how extensible the evaluation framework is to different model architectures. Although this would require additional experiments, the reviewer believes evaluating multiple MLLM families would provide stronger evidence of the benchmark's general applicability and allow future researchers to assess their own models with greater confidence.

Questions for authors

One concerned aspect is the complexity of the overall pipeline illustrated in Figure 4. An MLLM is already incorporated as a major component, and additional visual and language branches are built on top of it. The reviewer would like to better understand the design rationale and the strategies behind these architectural choices. Meanwhile, the change decoder, which appears to be one of the most important components of the proposed method, is illustrated only in a simplified manner. The reviwer believes a more detailed architectural illustration and a deeper explanation in the text would help readers better understand its role and functionality. In particular, aligning Figure 4 more closely with the descriptions in Section 4 would substantially improve the clarity and readability of the paper.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 8Wui

We thank the reviewer for the helpful comments. We reply to each point below.


W1. Design idea behind the pipeline (Fig. 4) and details of the change decoder

Thank you for this suggestion. We first explain the design idea and then list what we changed in the revision.

Why two branches on top of the MLLM? The two branches do not add a second backbone, and no SAM image encoder is used. They only read what the MLLM has already computed:

  • The visual change feature branch reads the vision tokens of the two views. It is needed because the generated text carries no pixel-level correspondence between two unregistered views.
  • The language branch reads the hidden states of the reasoning chain (CoT). With it, the decoder can use the target and the landmark named in the reasoning, not only a single [change] token.

Both are needed because an ECG target is defined by the relation between the two views together with the instruction. SAM-like decoders decode one image from one prompt token and stay at or below 28.2 mIoU, while our decoder reaches 35.6 (Table 3). Removing the cross-view alignment costs 2.0 points, and removing the CoT conditioning costs 0.7.

How the decoder works, in the order of Fig. 4:

  1. Attention pooling condenses the CoT hidden states into one CoT representation.
  2. The view head predicts which view contains the target. This view becomes the anchor view, so a missing object can be segmented in the reference view.
  3. Anchor-guided cross-view alignment lets each token of the anchor view find its matching content in the other view.
  4. Change fusion combines the anchor features, the aligned features, and their differences into the anchor-view change features.
  5. The change decoder then applies three steps: CoT grounding (a map that shows where the reasoning points to, supervised by the target mask), CoT injection (adds the CoT representation to the features where this map is high, and to the [change] embedding), and mask decoding (outputs the mask on the anchor view).

Changes in the revised paper (marked in blue):

  • Fig. 4 is updated. The change decoder block now lists its three steps, and the branch names and symbols match Sec. 4 exactly.
  • Sec. 4.1 is rewritten to walk through Fig. 4 module by module, using the same module names as the figure.
  • A new detailed architecture figure of the decoder is added (Fig. A10, App. D.1). It expands each module of Fig. 4 into its operations, and App. D.1 now describes them in the same order as the figure.

W2 & Q1. Only Qwen-based models are evaluated. Does the approach extend to other MLLMs?

This is not a limit of our approach, and Qwen is not special for this task. Our decoder only needs the vision tokens and the hidden states of the MLLM, so it can be attached to any MLLM that exposes them. To verify this, we attach it to two other MLLM families, InternVL3.5-8B and GLM-4.1V-9B, and run the same post-training.

Table A. Results on other MLLM families (mIoU, %).

Base MLLMMethodAddMissingReplaceRotateRelocateStateAVE
InternVL3.5-8BLISA + CoT-SFT23.520.425.824.119.249.627.1
InternVL3.5-8BECG-Grounder (CoT-SFT)27.623.832.427.322.754.131.3
InternVL3.5-8BECG-Grounder + VPPO30.926.535.232.624.157.334.4
InternVL3.5-8BECG-Grounder + SCPO (Ours)33.528.237.435.525.458.836.5
GLM-4.1V-9BLISA + CoT-SFT24.320.126.923.820.551.227.8
GLM-4.1V-9BECG-Grounder (CoT-SFT)28.923.534.027.124.055.632.2
GLM-4.1V-9BECG-Grounder + VPPO32.426.036.733.925.358.935.5
GLM-4.1V-9BECG-Grounder + SCPO (Ours)35.129.439.037.226.860.738.0

The trend is the same as on Qwen for both families (numbers are InternVL3.5-8B / GLM-4.1V-9B):

  • The change decoder helps: ECG-Grounder (CoT-SFT) is +4.2 / +4.4 AVE above LISA + CoT-SFT.
  • SCPO helps: it is +2.1 / +2.5 AVE above VPPO and +5.2 / +5.8 above the CoT-SFT checkpoint, and it is the best on all six change types.

The benchmark itself is model-agnostic. The evaluation only needs a predicted mask and the view it lies on, so any model that outputs a mask, or a box that a segmentation model turns into a mask, can be evaluated. Table 2 already includes InternVL3 and LLaVA-OV. In this rebuttal we also evaluate frontier proprietary MLLMs combined with SAM 3 (GPT, Gemini, and Claude; see our response to Reviewer R3, Q2), which reach 11.2–59.1 AVE. Both sets of results are now in the revised PDF (Table A10 in App. E.9; Table 2 and App. E.4), so that researchers working on other model families have direct reference points.

Reviewer_BsyEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Embodied Change Grounding (ECG): given a reference image, a query image taken after the scene changed, and an instruction referring to a change, a model must segment the changed object in whichever image contains it. It contributes ECG-Bench (10K samples from 1,859 simulated, industrial, and real scenes, covering six change types) and ECG-50K, a scene-disjoint simulated training set with Change-CoT reasoning traces. The proposed ECG-Grounder generates a reasoning chain with an MLLM, then decodes the mask with a CoT-guided cross-view decoder that selects the target view and aligns the two images. It is post-trained with SCPO, which adds a GRPO reward term favoring chains that are more sensitive to masking the changed object and its landmark than to masking a matched nuisance region. The full model reaches 35.6/38.7 AVE on two backbones, ahead of matched RL baselines such as VPPO.

Critical strengths
  1. ECG introduces a useful task and benchmark combining language-referred changes, two unaligned views, and targets that may appear in either image. Existing change detection and referring segmentation tasks do not fully cover this setting.

  2. The paper is well-written and presented. Reading it is very smooth.

Critical weaknesses
  1. Limited methodological novelty. SCPO adds a single reward term to standard GRPO. Removing the nuisance subtraction yields a PAPO-style reward, with AVE only 1.2 lower than the full method (Table 4).

  2. Selectivity does not establish faithful reasoning. The critical mask removes both the object and its landmark (§4.2). Moreover, Fig. A12 evaluates selectivity using the same JS terms that define the reward. Neither provides an independent assessment of whether the model’s reasoning is faithful.

  3. The need for two views is not established. Instructions always provide a spatial anchor (App. B.3), yet the paper lacks a single-view baseline with a matched compute budget. Table 3 varies only the decoder and therefore does not isolate the contribution of the second view.

  4. Real-scene localization remains unevaluated. Of the 10K samples, 9K are simulated. Although the remaining 1K real samples are presented as a synthetic-to-real test (§5.1), no localization results are reported for this split.

Other issues (these do not affect my rating)

  • Overlap assumption. Without sufficient overlap, an absent object cannot be distinguished from one outside the field of view. Real image pairs with large viewpoint differences were excluded (App. B.1), and Fig. A12 evaluates yaw differences only up to 45°.

  • Dependence on simulator annotations. Constructing SCPO’s critical mask requires object and landmark masks supplied by the simulator (§4.2). Applying the same training procedure directly to real revisits would therefore require additional annotations.

  • Mask resolution. Masks are predicted at a fixed resolution of 192×256 (App. D.1), and segmentation of small real objects remains incomplete, with IoUs of 0.29 and 0.31 (App. G). Please clarify the resolution at which mIoU is computed.

Questions for authors
  1. How is RgroundR_{\mathrm{ground}} (Eq. 5) defined, and how are wrong-view masks scored in the reward and in mIoU? Unpenalized wrong views would change how I read Table 2.
  2. How does a budget-matched single-view model perform, split by synthetic/real and by viewpoint overlap? Near parity would weaken the cross-view claim.
  3. On random samples checked independently, does SCPO reduce wrong change types or unsupported claims vs. VPPO? Please also mask target and landmark separately.
4 comments
AuthorsOpenReview ↗

Response to Reviewer BsyE (Part 1/3)

We thank the reviewer for the careful and constructive comments. We reply to each point below. All new results are now in the revised PDF (changes in blue).


Q1. Definition of R_ground (Eq. 5) and how wrong-view masks are scored

Thank you for pointing this out. The grounding reward is

R_ground = 1[a = a*] · IoU(M, M*)

where a and a* are the predicted and ground-truth views, M and M* are the predicted and ground-truth masks, and 1[·] equals 1 when the two views agree and 0 otherwise. A mask on the wrong view gets zero reward, no matter how good the mask is.

mIoU uses the same rule: a prediction on the wrong view counts as IoU = 0. R_view is only a small extra reward for choosing the correct view. So wrong views are penalized in both the reward and the metric, and all numbers in Table 2 already include this penalty. Both are now stated in the revised PDF, next to Eq. 5 and in Sec. 5.1.


Q2 & W3. Is the second view needed? Single-view and mismatched-pair tests

We agree that Table 3 only varies the decoder and does not isolate the second view. We therefore ran the test the reviewer asks for. We keep each trained model, the instruction, and the inference settings fixed, and change only the visual input:

  • Reference only / Query only: the model sees only one of the two images.
  • Mismatched pair: we keep the image that holds the ground-truth mask and replace the other image with the same-role image (reference for reference, query for query) from a different scene of the same data source. Using the same source keeps the image style and resolution unchanged, so the model cannot notice the swap from style alone.

Table A. AVE (%) on ECG-Bench under different visual inputs (Qwen2.5-VL-7B). Brackets: relative drop from the full pair of the same model. GRPO is the VisionReasoner† row of Table 2.

MethodReference onlyQuery onlyMismatched pairFull pair
ECG-Grounder (CoT-SFT)6.1 (−80%)15.5 (−49%)13.3 (−56%)30.3
+ GRPO4.2 (−87%)11.3 (−65%)14.0 (−57%)32.6
+ PAPO4.1 (−88%)8.9 (−73%)12.5 (−63%)33.5
+ VPPO4.5 (−87%)7.1 (−79%)12.8 (−62%)33.9
+ SCPO (Ours)3.9 (−89%)7.5 (−79%)14.6 (−59%)35.6
Claude Haiku 5.5 + SAM 33.8 (−90%)6.3 (−84%)16.5 (−58%)39.3

Table B. SCPO on different evaluation splits (AVE, %).

Evaluation splitReference onlyQuery onlyFull pair
Synthetic4.87.836.4
Real3.65.428.3
Low overlap3.911.430.9
Medium overlap3.66.434.8
High overlap4.37.036.6
All3.97.535.6

What the results show:

  1. Every model is substantially lower without the correct second view. AVE falls from 30.3–35.6 (full pair) to 3.9–15.5, a relative drop of 49–89%.
  2. The drop is not specific to our model. A strong frontier pipeline (Claude Haiku 5.5 predicts the box, SAM 3 segments it) falls from 39.3 to 3.8 (−90%) with the reference only, to 6.3 (−84%) with the query only, and to 16.5 (−58%) with a mismatched pair.
  3. There is no near parity on any split (Table B). On the synthetic split, the real split, and at every level of viewpoint overlap, single-view input reaches at most 11.4 AVE, while the full pair reaches at least 28.3. The single-view results show no clear trend across overlap levels: with only one image the model mostly has to guess, and it is sometimes right by chance, so these small numbers fluctuate.
  4. The location words in the instruction (e.g., "on the lower-right floor") cannot locate the target without the second view. If they could, the single-view results would stay close to the full-pair results, but they fall by 49–89%. The small accuracy that remains comes mostly from change types that are partly visible in one image, for example a missing object in the reference view or a state change in the query view.
  5. The composition of ECG-Bench explains why. We counted, for each split, how many samples come from image pairs with two or more changes, and how many have other objects of the same category as the target in the scene (Table C). Both cases are the majority. In such scenes, the instruction and one image do not single out the target: only a comparison of the two views tells which object is the referred change.

Table C. Statistics of ECG-Bench samples (our count).

SplitSamples from pairs with ≥2 changesMax. changes in one pairSamples with same-category distractors
Real85%1058%
Synthetic88%696%

These results show that ECG-Bench needs cross-view reasoning. They are now in the revised PDF: Table 5 and Sec. 5.4 in the main text, and App. E.5 with the per-type and per-split results (Tables A4 and A5). We also added two visualized mismatched-pair cases (Figs. A15 and A16).

AuthorsOpenReview ↗

Response to Reviewer BsyE (Part 2/3)

Q3 & W2. Independent check of reasoning faithfulness; masking target and landmark separately

We agree that Fig. A12 uses the same JS terms as the reward, so it is not an independent test. We therefore add a human-verified evaluation that does not use any JS term.

(1) Unsupported claims. An unsupported claim is a factual statement in the CoT that the images do not support. We randomly sample 500 real-world examples and generate CoTs with the original inference prompts. Model names are hidden. ChatGPT Sol gives a first assessment, and five trained human annotators then check every assessment by hand. We report:

  • Hallucination rate: CoTs with at least one wrong visual claim / all checked CoTs.
  • Per-claim error rate: wrong claims / all claims.

(2) Wrong change types. The original CoT does not always name the change type. So we add one line to the prompt that asks each model to first state the change type, and compare it with the ground-truth label. The rate is computed over the CoTs that state a type.

Table D. Independent evaluation of the reasoning chains (500 random real-world samples; lower is better).

MetricGRPOPAPOVPPOSCPO (Ours)
Hallucination rate64.0%49.4%45.8%38.7%
Claims per CoT4.552.913.012.33
Per-claim error rate22.9%24.3%21.4%18.7%
Wrong change-type rate21.7%22.6%20.9%18.5%

SCPO is the best on every metric. Compared with VPPO, it lowers the hallucination rate from 45.8% to 38.7%, the per-claim error rate from 21.4% to 18.7%, and the wrong change-type rate from 20.9% to 18.5%. The lower per-claim error rate shows that the gain does not come only from writing shorter chains: PAPO also shortens the chain, but its per-claim error rate does not go down.

In the revised PDF, these results are in Sec. 5.4 and App. E.6 (Table A6). We also added visualizations of the annotation: Fig. 7 in the main text shows one annotated GRPO / SCPO pair, and Figs. A17 and A18 show six more, with every unsupported claim marked and explained.

(3) Masking target and landmark separately. As suggested, we train SCPO with the critical mask covering the target only, the landmark only, or both. All other settings are unchanged.

Table E. Ablation on the critical mask (AVE, %).

Critical maskAVE ↑ΔAVE
Plain GRPO (no critical mask)32.6−3.0
Target only34.8−0.8
Landmark only33.4−2.2
Target + Landmark (Ours)35.6–

Masking the target gives most of the gain (+2.2 over GRPO). Masking the landmark alone also helps (+0.8). Masking both is the best (+3.0), so the two are complementary: the instruction refers to the target through its landmark, and the reasoning should depend on both. The two new rows are now in Table 4 of the revised PDF.


W1. Limited methodological novelty

We would like to clarify two points.

First, the 1.2-point gain in Table 4 is only the effect of the nuisance subtraction. The full SCPO improves AVE by 3.0 points over GRPO. The difference from a PAPO-style reward is what the model is rewarded for: a PAPO-style reward encourages the model to depend on the visual input in general, while SCPO rewards sensitivity to the change-critical evidence and, at the same time, penalizes sensitivity to unrelated visual variation.

Second, SCPO also makes the reasoning shorter and faster, not only more accurate.

Table F. Accuracy and reasoning cost (Qwen2.5-VL-7B backbone).

MethodAVE ↑Avg. CoT tokens ↓Latency ↓Token reduction vs. GRPO ↑
CoT-SFT30.3110.13.45 s–
GRPO32.690.72.37 s–
PAPO33.547.71.52 s47.4%
VPPO33.948.81.69 s46.2%
SCPO (Ours)35.640.61.31 s55.2%

Compared with GRPO, SCPO uses 55.2% fewer CoT tokens and has 44.7% lower latency. Compared with PAPO, it gains +2.1 AVE with 14.9% fewer tokens and 13.8% lower latency. The token and latency numbers are now in Table 6 (Sec. 5.4) of the revised PDF. Together with Figs. 5–6, the lower hallucination rate (Table D), and the larger gain on real scenes (Table G in Part 3), these results show that the selective reward changes how the model reasons across views, beyond a small gain in accuracy.

AuthorsOpenReview ↗

Response to Reviewer BsyE (Part 3/3)

W4. Real-scene localization is not reported

We now report results on each subset of ECG-Bench separately. The real-world subset (993 samples) is a pure synthetic-to-real test, because ECG-50K has no real-world pairs.

Table G. mIoU (%) on each subset of ECG-Bench (Qwen2.5-VL-7B backbone).

MethodSimulated (7K)Industrial (2K)Real-world (1K)All
Qwen2.5-VL-7B + SAM 2 (training-free)9.49.09.39.3
LISA23.420.418.622.3
ECG-Grounder (CoT-SFT)30.432.425.430.3
+ GRPO32.935.625.232.6
+ PAPO33.637.125.833.5
+ VPPO34.137.425.133.9
+ SCPO (Ours)35.639.328.335.6
  • SCPO is the best on real scenes: 28.3, which is +3.1 over GRPO, +3.2 over VPPO, and +2.5 over PAPO.
  • Other RL methods gain mainly on simulated data. GRPO and VPPO improve the simulated and industrial subsets but not the real one (25.2 and 25.1 vs. 25.4 for CoT-SFT). SCPO is the only RL method with a clear gain on real scenes (+2.9 over CoT-SFT).
  • Our decoder also transfers better than LISA: 25.4 vs. 18.6 on real scenes.

Real scenes are still the hardest subset. This split is now reported in the revised PDF: Table 6 (Sec. 5.4) in the main text and Table A7 (App. E.7) for all methods.


Minor W1. Overlap assumption

Thank you for this comment. We would like to clarify that ECG requires enough visual evidence for the referred change in the given pair. It does not require a high overall overlap between the two fields of view.

A large viewpoint difference and an unobservable change are two different cases. With a large viewpoint difference, the model can still locate the change by cross-view matching, as long as the target and its surrounding region are visible. But if the changed region is fully outside one camera view, the two images alone cannot tell whether the object is really gone or just not captured. This ambiguity comes from the input, and no cross-view method can remove it.

For this reason, ECG-Bench keeps only pairs in which the change can be verified from the images. The goal is to keep the task well defined, not to avoid large viewpoint changes. This scope is now stated in the Limitations (App. H) of the revised PDF, and the caption of the yaw analysis (Fig. A13 there) gives its 45° range.


Minor W2. Dependence on simulator annotations

We agree. If SCPO is trained directly on unlabeled real revisits, it needs extra masks for the target and the landmark.

Our setting is to train on simulated data, where these masks are exact and free. This is one main reason for our training strategy. The masks are needed only during training: at test time the model runs on real scenes without any extra annotation. For future training on unlabeled real data, pseudo masks are one way to reduce this cost. This limitation is now stated in App. H of the revised PDF.


Minor W3. Mask resolution

All mIoU values are computed at each image's native resolution, not at 192×256. The predicted 192×256 mask logits are bilinearly upsampled to the original size of the annotated view (e.g., 512×512 for ChangeSim, up to 1904×1428 for our real-world captures) and thresholded at 0.5. IoU is then computed against the full-resolution ground-truth mask and averaged over samples. All baselines use the same protocol: their masks are mapped back to the native resolution before scoring.

We agree that the fixed decoder resolution limits boundary accuracy for small objects in high-resolution real images. This partly explains the low IoUs in App. G. The protocol is now stated in Sec. 5.1 and App. D.1 of the revised PDF, and the fixed decoder resolution is listed as a limitation in App. H.

Thanks for the detailed response. It clears up most of what I asked about. The wrong-view scoring is now clear, the real-split numbers look good, and the single-view and mismatched-pair tests make a convincing case for the second view. I still think SCPO itself is a modest contribution though it is proven to be helpful: it adds one reward term to GRPO, and the gain over VPPO is under 2 points. Most of its token reduction is already achieved by PAPO and VPPO. However, the proposed task and benchmark are strong contributions in their own right. I therefore lean toward acceptance and keep my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.