- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces Embodied Change Grounding (ECG): given two photos of the same scene and an instruction, segment the object that changed. It presents a benchmark with 10k samples, and a 50k training set. The proposed method selects which view holds the target, aligns the two views, and decodes the mask from its reasoning. The authors further introduce SCPO, an RL reward that favors reasoning that depends on the region that actually changed, rather than on unrelated parts of the image. Experiments show that existing MLLMs and standard RL perform poorly on ECG, while ECG-Grounder with SCPO achieves the best results.
Critical strengths
ECG is a well-motivated problem that goes beyond standard change detection. The benchmark is also reasonably diverse, covering multiple types, scenes, and viewpoint changes.
The cross-view decoder explicitly models correspondence between the two views, and SCPO encourages reasoning that is grounded in the relevant change evidence. The method is further supported by extensive ablations and transfer experiments.
Critical weaknesses
The statistical robustness of the reported improvements is unclear. Although SCPO achieves the best overall results, its margins over the strongest closely matched baseline, VPPO, are relatively modest (+1.7 and +2.1). Some component-level gains are smaller still: the matched nuisance branch contributes +1.2 and entropy-based token selection only +0.5. The paper does not report results across multiple random seeds, repeated-run variance, confidence intervals, or statistical significance tests for these comparisons. Given the stochasticity of RL post-training, it is difficult to determine whether these relatively small margins are robust or partly due to run-to-run variation.
Questions for authors
-
The paper states that ECG-50K contains no real-world pairs and that the real split of ECG-Bench therefore measures synthetic-to-real transfer. Could the authors report performance separately on simulated, industrial, and real-world subsets?
-
Could the authors evaluate a frontier MLLM (e.g. GPT, Gemini, etc.) + state-of-the-art segmentation model (e.g. SAM 3) baseline on ECG-Bench? This would provide a stronger reference point for assessing the necessity of the proposed ECG-specific architecture and training.
-
How does the model behave when multiple changes occur simultaneously, or when no relevant change is present? These cases would test whether the model can handle more realistic change patterns rather than assuming exactly one target change per pair.
-
How does the trained model perform when given the instruction with only the query image, or only the reference image, or a mismatched reference-query pair? Some instructions may contain strong cues about the answer, for example by naming the change type and location ("the newly appeared object on the lower-right floor"). If single-view and mismatched-pair performance is substantially lower than the full two-view setting, this would provide stronger evidence that ECG-Bench genuinely requires cross-view reasoning rather than allowing the model to exploit linguistic or single-image shortcuts.
6 comments
Response to Reviewer xmbu (Part 2/3)
Q3. Multiple simultaneous changes, and cases with no relevant change
(1) Multiple changes. ECG-Bench already covers this case to a large extent.
Table D. Multi-change statistics of ECG-Bench.
| Split | Samples whose pair has ≥2 changes | Max. changes in one pair | Samples with same-category distractors |
|---|---|---|---|
| Real | 85% | 10 | 58% |
| Synthetic | 88% | 6 | 96% |
Most samples (85% of the real split and 88% of the synthetic split) come from pairs with two or more changes, and the rest have a single change. We also split the real part by the number of changes:
Table E. Results on the real part, split by the number of changes (mIoU, %).
| Real part | Share | GRPO | PAPO | SCPO (Ours) |
|---|---|---|---|---|
| Pairs with ≥2 changes | 85% | 24.1 | 24.7 | 27.1 |
| Pairs with a single change | 15% | 31.4 | 32.4 | 35.5 |
| All | 100% | 25.2 | 25.8 | 28.3 |
Referring to one change among several is clearly harder for every method (24.1–27.1 vs. 31.4–35.5). SCPO is the best in both groups, and on multi-change pairs it is +3.0 over GRPO and +2.4 over PAPO. Table E is now Table A8 (App. E.7) of the revised PDF.
(2) No relevant change. ECG-Bench currently follows the target-present setting: every instruction refers to a change that really exists. This is the common setting in reasoning segmentation and grounding (e.g., LISA, Seg-Zero, VisionReasoner). It tests whether a model can locate and segment the target under a complex instruction, not whether the target exists.
We agree with the reviewer that in real embodied use the described change may not have happened, so this ability is worth testing. We therefore build 2,400 negative instructions at two difficulty levels (1,200 each, about 24% of the benchmark size):
- N0 (unrelated instruction): the instruction is borrowed from the other domain (warehouse ↔ home), and we make sure that its landmark does not exist in the current scene. It tests whether the model rejects a clearly unrelated instruction.
- N1 (hard negative): we keep the location and the landmark of the original instruction, but describe a change that did not happen, by flipping the change direction (polarity flip) or swapping the change type (type swap). Every negative is checked against all annotated changes of the pair.
Step 1: no extra training. We first test CoT-SFT, GRPO, PAPO, and SCPO as they are. Even when the prompt explicitly asks the model to output NO_CHANGE if no change matches, all four models still tend to output a mask. The AUROC of their rejection scores is only 0.46–0.59, which is close to chance. So changing the prompt alone does not give reliable rejection, and this is a limit of target-present training.
Step 2: training with negatives. We then add the two types of negative samples to the training set and retrain CoT-SFT, GRPO, VPPO, and SCPO under the same settings. We check (i) how often the model correctly rejects a negative instruction (N0 / N1 Acc.), (ii) how often it wrongly rejects a positive one (False Rej.), and (iii) whether it keeps its grounding accuracy on the original benchmark.
Table F. Results after training with negative instructions.
| Method | Original mIoU ↑ | Retrained mIoU ↑ | Δ | N0 Acc. ↑ | N1 Acc. ↑ | False Rej. ↓ |
|---|---|---|---|---|---|---|
| CoT-SFT | 30.3 | 29.5 | −0.8 | 76.8 | 32.4 | 6.9% |
| GRPO | 32.6 | 31.5 | −1.1 | 75.1 | 39.8 | 5.4% |
| VPPO | 33.9 | 31.6 | −2.3 | 78.2 | 33.4 | 8.0% |
| SCPO (Ours) | 35.6 | 34.3 | −1.3 | 79.5 | 40.8 | 5.5% |
- With negative samples in training, all methods learn to reject unrelated instructions (N0 Acc. 75.1–79.5%), at a small cost in grounding accuracy (−0.8 to −2.3 mIoU).
- SCPO gives the best trade-off: the highest rejection accuracy on both N0 and N1, the highest mIoU after retraining (34.3), and a low false rejection rate (5.5%).
- Hard negatives remain difficult for all methods (N1 Acc. 32.4–40.8%). Deciding whether a specific change really happened needs fine cross-view comparison, and we see this as an open challenge; it is listed as a limitation in App. H.
These results are now in App. E.8 of the revised PDF (Table A9). We also added a visualization (Fig. A19) with four real cases of SCPO: a rejected and a non-rejected instruction for each of N0 and N1, with the reasoning chain of each. We will release the negative set as an extension of ECG-Bench.
Response to Reviewer xmbu (Part 3/3)
Q4. Single-view input and mismatched reference–query pairs
We ran all three tests. We keep each trained model and the instruction fixed and change only the visual input:
- Reference only / Query only: the model sees only one of the two images.
- Mismatched pair: we keep the image that holds the ground-truth mask and replace the other image with the same-role image (reference for reference, query for query) from a donor sample. The donor comes from the same data source but a different scene, and shares no image file with the original sample. Using the same source keeps the image style and resolution unchanged, so the model cannot notice the swap from style alone.
In the tables below, "Drop in AVE" and the values in brackets are measured against the full-pair result of the same model (relative drop in %). In the revised PDF, Tables G and I are Table A4 (App. E.5) and Table H is Table 5 (Sec. 5.4).
Table G. SCPO under different visual inputs (mIoU, %, Qwen2.5-VL-7B).
| Input | Add | Missing | Replace | Rotate | Relocate | State | AVE | Drop in AVE |
|---|---|---|---|---|---|---|---|---|
| Reference only | 0.7 | 16.3 | 0.7 | 1.6 | 0.7 | 3.1 | 3.9 | −31.7 (−89%) |
| Query only | 4.5 | 0.6 | 6.3 | 5.8 | 4.1 | 23.4 | 7.5 | −28.1 (−79%) |
| Mismatched pair | 13.1 | 14.5 | 13.7 | 15.6 | 9.4 | 21.1 | 14.6 | −21.0 (−59%) |
| Full pair | 33.8 | 27.1 | 36.2 | 35.6 | 23.4 | 57.7 | 35.6 | – |
Table H. All post-training methods under different visual inputs (AVE, %).
| Method | Reference only | Query only | Mismatched pair | Full pair |
|---|---|---|---|---|
| ECG-Grounder (CoT-SFT) | 6.1 (−80%) | 15.5 (−49%) | 13.3 (−56%) | 30.3 |
| + GRPO | 4.2 (−87%) | 11.3 (−65%) | 14.0 (−57%) | 32.6 |
| + PAPO | 4.1 (−88%) | 8.9 (−73%) | 12.5 (−63%) | 33.5 |
| + VPPO | 4.5 (−87%) | 7.1 (−79%) | 12.8 (−62%) | 33.9 |
| + SCPO (Ours) | 3.9 (−89%) | 7.5 (−79%) | 14.6 (−59%) | 35.6 |
Table I. Claude Haiku 5.5 + SAM 3 under different visual inputs (mIoU, %).
| Input | Add | Missing | Replace | Rotate | Relocate | State | AVE | Drop in AVE |
|---|---|---|---|---|---|---|---|---|
| Reference only | 1.7 | 11.0 | 3.5 | 3.6 | 0.9 | 2.2 | 3.8 | −35.5 (−90%) |
| Mismatched pair | 14.4 | 13.1 | 18.5 | 13.4 | 18.6 | 21.2 | 16.5 | −22.8 (−58%) |
| Full pair | 38.5 | 31.5 | 43.7 | 27.0 | 38.5 | 56.6 | 39.3 | – |
What the results show:
-
Single-view and mismatched-pair results are substantially lower than the full two-view setting, so ECG-Bench needs cross-view reasoning. Every method drops from 30.3–35.6 AVE to 3.9–15.5, a relative drop of 49–89%: 80–89% with the reference only, 49–79% with the query only, and 56–63% with a mismatched pair. The frontier pipeline (Claude Haiku 5.5 + SAM 3) drops in the same way, from 39.3 to 3.8 (−90%), 6.3 (−84%), and 16.5 (−58%).
-
A few change types keep some signal from one image. With a single image, almost every type falls to a very low level. The exceptions are expected: a missing object is visible in the reference view (Missing, 16.3 with the reference only), and a state change is often visible in the query view (State, 23.4 with the query only). With a mismatched pair, some types also do not fall to near zero, because the image that holds the target sometimes gives hints about it. So some samples can be roughly guessed from the image that holds the target. Even in this setting, every model loses more than half of its AVE (56–63%), and all types stay far below the full-pair results.
-
Many of the remaining correct cases are lucky guesses. Figs. A15 and A16 (App. E.5) of the revised PDF show mismatched-pair cases where the prediction is still correct. In a part of them, the model hallucinates a change and happens to pick the right object. Locating the target reliably still needs cross-view reasoning.
-
Our instructions are disambiguated to limit language shortcuts. A precise reference such as "the newly appeared object on the lower-right floor" is used when the pair has several changes, distractors, or objects of the same category, so the text alone does not pick out the target. When the pair has only one change, we give only a coarse description of the change. This agrees with the statistics in Q3: 85% of real samples and 88% of synthetic samples come from pairs with two or more changes, and 58% / 96% contain distractors of the same category. This procedure is now described in App. E.5.
Response to Reviewer xmbu (Part 1/3)
We thank the reviewer for the careful and constructive comments. We ran new experiments for each point and report them below. All new results are in the revised PDF (in blue).
W1. Statistical robustness of the reported gains
We train GRPO (the VisionReasoner† row of Table 2), VPPO, and SCPO three times each with different seeds, on both Qwen2.5-VL-7B and Qwen3.5-9B, under identical settings.
Table A. Results over three random seeds (AVE, %, mean ± std).
SCPO has the highest mean and the lowest standard deviation on both backbones. Its gains over VPPO (+1.8 and +2.2) are several times larger than the standard deviations (0.2–0.4), so they are unlikely to be run-to-run variation.
We also repeat the two component ablations with three seeds (Qwen2.5-VL-7B).
Table A2. Component ablations: AVE (%) over three random seeds, and average CoT length.
The seed means stay within 0.1 of the numbers in Table 4. The gain from the nuisance branch (+1.2) is four times the largest standard deviation (0.3), so it is robust. The gain from entropy selection in AVE (+0.5) is consistent across seeds but small. Both tables are now Table A11 (App. E.10) of the revised PDF.
Entropy selection mainly affects the length of the reasoning (last column of Table A2). Without it, the average CoT length rises from 40.6 to 49.5 tokens, about the level of PAPO (47.7) and VPPO (48.8), while removing the nuisance branch leaves it almost unchanged (41.5). We believe the reason is that the reward is measured only at the few tokens where the model makes its decisions, so extra descriptive text earns no reward.
Q1. Results on the simulated, industrial, and real-world subsets
We report each subset separately below (now Table 6 and Table A7 of the revised PDF). The real-world subset (993 samples) is a pure synthetic-to-real test, because ECG-50K has no real-world pairs.
Table B. mIoU (%) on each subset of ECG-Bench (Qwen2.5-VL-7B backbone).
Q2. Frontier MLLM + state-of-the-art segmentation model
We evaluate seven proprietary MLLMs combined with SAM 3 (now Table 2 and App. E.4 of the revised PDF).
Table C. Frontier MLLM + SAM 3 on ECG-Bench (mIoU, %).
These results support ECG-specific training: the same Qwen3.5-9B reaches only 16.0 AVE without training (Table 2) and 38.7 after our post-training.