- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Erase-to-Verify, a training-free agentic self-correction framework for Referring Image Segmentation (RIS). The central idea is to treat a predicted segmentation mask as a hypothesis that can be verified through visual intervention. Specifically, the framework applies localized blurring to the predicted mask region and examines whether the referred target remains visible in the modified image. If residual target evidence is detected, the framework identifies the failure type and performs targeted refinement. Experiments on the RefCOCO-family benchmarks and ReasonSeg demonstrate its effectiveness.
Critical strengths
-
Novel and Intuitive Verification Perspective. The paper introduces an interesting perspective by treating predicted segmentation masks as hypotheses to be verified through visual intervention rather than relying solely on direct mask inspection. This idea is intuitive and well-motivated, offering a new direction for improving the reliability of training-free referring image segmentation.
-
Experimental Performance and Supporting Ablation Studies. The proposed framework achieves consistent improvements across all eight RefCOCO-family benchmark splits. Moreover, the component ablation studies show that target-suppressed verification provides additional improvements beyond iterative refinement based on mask overlays, offering empirical support for the effectiveness of the proposed mechanism.
Critical weaknesses
-
Limitations of the Core Verification Mechanism. The proposed framework determines mask validity based on whether the referred target disappears after suppressing the predicted region. However, target disappearance primarily reflects target coverage and does not necessarily guarantee accurate segmentation boundaries. For example, an over-segmented mask may completely suppress the target while including substantial background regions. Although the authors acknowledge this limitation, the paper lacks quantitative analysis of the verification mechanism under different segmentation error types.
-
Insufficient Analysis of Computational Efficiency and Refinement Stability. Although the paper reports inference time comparisons, these results are not obtained under unified experimental conditions. Furthermore, the computational costs of individual stages, the number of VLM and SAM3 calls, and the stability of predictions across refinement iterations are not sufficiently analyzed.
-
Limited Fairness of Model Comparisons on ReasonSeg. The ReasonSeg experiments compare the proposed method using Qwen3-VL-4B against competing methods based on Qwen2.5-VL-3B or other model configurations. Although these models have relatively similar parameter scales, differences in their underlying capabilities may influence the reported performance.
Questions for authors
-
Reliability of the Verification Mechanism. How reliably does the proposed Erase-to-Verify mechanism distinguish accurate masks from incorrect ones, particularly in cases of over-segmentation or small residual target regions? Such evidence would help assess the reliability of the core verification mechanism.
-
Computational Efficiency and Refinement Stability. Could the authors provide a more detailed analysis of the computational overhead, including the average number of VLM and SAM3 calls, the inference time of individual stages, and comparisons under unified hardware and inference settings? Additionally, how frequently does iterative refinement lead to performance degradation rather than improvement? These results would help clarify the efficiency and stability of the proposed framework.
-
Fairness of Comparisons on ReasonSeg.
In the ReasonSeg experiments, the proposed method uses Qwen3-VL-4B, whereas the competing methods employ different VLM backbones. Could the authors provide comparisons using the same VLM backbone? This would help determine whether the performance gains can be attributed primarily to the proposed framework.
5 comments
Weakness 3 (Limited fairness of the model comparison on ReasonSeg). We agree. The original Table 2 compared our Qwen3-VL-4B system with baselines using Qwen2.5-VL-3B, whose underlying capabilities are not equivalent. In addition, the Tarot-SAM3 numbers in the original Table 2 were taken from its arXiv version, which has since been withdrawn. We therefore used the official code its authors previously released on GitHub and re-ran Tarot-SAM3 in the same environment as our method, with the same evaluation pipeline, under two backbones, Qwen2.5-VL-3B (the backbone used by the baselines in the original Table 2) and Qwen2.5-VL-7B, and compared it with Erase-to-Verify:
| Method (same environment) | Backbone | Val gIoU | Val cIoU | Test gIoU | Test cIoU |
|---|---|---|---|---|---|
| Tarot-SAM3 (reproduced) | Qwen2.5-VL-3B | 57.2 | 38.1 | 57.3 | 44.6 |
| Erase-to-Verify | Qwen2.5-VL-3B | 63.7 | 53.9 | 57.8 | 50.1 |
| Tarot-SAM3 (reproduced) | Qwen2.5-VL-7B | 67.8 | 56.8 | 67.8 | 63.4 |
| Erase-to-Verify | Qwen2.5-VL-7B | 70.4 | 60.9 | 67.6 | 66.6 |
With the same backbone and environment, we lead in seven of the eight comparisons (two backbones × four metrics); the only exception is test gIoU at 7B, where the two are essentially tied (67.6 vs. 67.8). At 3B we lead on all four metrics (val gIoU +6.5, val cIoU +15.8, test gIoU +0.5, test cIoU +5.5); at 7B we lead on three (val gIoU +2.6, val cIoU +4.1, test cIoU +3.2). The advantage is larger at 3B; a plausible explanation is that a weaker VLM makes more errors in its first-pass prediction, leaving more room for erase-to-verify and repair to correct them. Moreover, we use only a frozen VLM, SAM3 and a fixed blur operator, whereas Tarot-SAM3 additionally uses a frozen DINOv3 (ViT-B) to detect over-/under-segmentation. For the reproduction, we ran its official code exactly with the settings it specifies; because it was re-run in a unified environment, Tarot-SAM3's numbers differ from those originally reported, and our focus is the relative comparison within the same environment. The revised Table 2 will be centered on same-backbone, same-environment controlled comparisons and will remove the cross-backbone +5.5/+11.0 claim on val.
Q1 (How reliable is the verification mechanism in distinguishing accurate from incorrect masks, especially for over-segmentation or small residual regions?)
On the same 10,786 verification decisions as in Weakness 1, the overall decision quality is: Accept precision 80.1%, recall of good masks 69.9%, F1 74.7%. Over-segmented masks are accepted 71.4% of the time, above the 59.6% for bad masks in general, but they make up only about 9.4% of all false accepts.
Regarding target size, the table below reports the first-pass mask accuracy (IoU > 0.5) for different target sizes:
| Target size (share of image area) | Share of samples | First-pass mask accuracy (IoU > 0.5) | Accept precision |
|---|---|---|---|
| Small (< 5%) | 22.1% | 81.2% | 81.64% |
| Medium (5%–10%) | 39.7% | 86.3% | 88.38% |
| Large (> 10%) | 38.2% | 91.2% | 93.03% |
Two points are worth noting. First, Accept precision is higher than the first-pass mask accuracy in all three size bins, showing that the verifier acts as a filter for every size: the masks it accepts are of above-average quality, rather than everything being accepted. Second, the spread in Accept precision across the three bins (81.64% vs. 93.03%, about 11 points) is about the same as the spread in first-pass accuracy (81.2% vs. 91.2%, about 10 points), showing that the verifier introduces no additional systematic bias against small targets; its reliability varies with target size in the same way as the difficulty of the first-pass prediction itself. The lower Accept precision on small targets (81.64%) is fundamentally because small targets are harder to segment correctly, which also leaves more room for improvement; we will state this plainly as a known limitation in the revision.
Q2 (Detailed analysis of computational overhead and optimization stability). Please see the per-stage call-count table and the stability analysis under Weakness 2. About 67.3% (RefCOCO) to 77.6% (ReasonSeg) of the samples are accepted in the first round, stopping after the initial proposal and one verification; re-localization and zoom/point repair are triggered only when the verifier returns Refine.
Q3 (Same-backbone comparison on ReasonSeg). We answer from two angles.
First, a comparison with exactly the baselines' backbones. In the same environment, against reproduced Tarot-SAM3 (full table under Weakness 3): with Qwen2.5-VL-3B we lead on all four metrics (val gIoU +6.5, val cIoU +15.8, test gIoU +0.5, test cIoU +5.5); with Qwen2.5-VL-7B we lead on val gIoU, val cIoU and test cIoU by +2.6, +4.1 and +3.2, and test gIoU is essentially tied (67.6 vs. 67.8).
Second, the gain from the framework itself with the backbone fixed. Table 3 gives controlled results on ReasonSeg on our own stack (Qwen3-VL-4B + SAM3):
| Variant | RSeg Val gIoU | RSeg Test gIoU |
|---|---|---|
| Base (single pass) | 69.1 | 63.3 |
| + Iteration (Overlay) | 70.7 | 65.4 |
| Erase-to-Verify (full) | 73.0 | 67.8 |
The full system improves over the single-pass prediction by +3.9/+4.5 gIoU and over the iteration-only variant by +2.3/+2.4 gIoU, which isolates the gain brought by the framework itself. Both the ReasonSeg Base (69.1/63.3) and the full system (73.0/67.8) in Table 3 use Qwen3-VL-4B + SAM3; the full-system numbers match Table 2, and Base is the single-pass prediction on the same stack, so the two can be compared directly.
Together, the two sets of results show that the improvement on ReasonSeg comes mainly from the framework itself, not just from a stronger backbone. The revised Table 2 will be centered on same-backbone, same-environment controlled comparisons, state the backbone of each row in the caption, and remove the cross-backbone +5.5/+11.0 claim.
Thank you again for your review. Your questions on verifier reliability and on cost and stability led us to evaluate the verifier directly and to report its shortcomings honestly.
Thank you for the careful review. We first respond to the three key weaknesses and then answer the three questions.
Weakness 1 (The verification mechanism reflects coverage rather than boundary accuracy). We agree, and Appendix E says the same: target disappearance after blurring only tests whether any target evidence remains outside the mask; it does not penalize extra area, so an over-segmented mask can pass the test. On 10,786 verification decisions on RefCOCO val (IoU >= 0.75 counts as a good mask), we measured how the verifier handles over-segmented masks:
Over-segmented masks are accepted more often (71.4%) than bad masks in general (59.6%), which is consistent with the design expectation that the erase test checks coverage and does not penalize boundaries; this is a known limitation. However, over-segmented masks make up only about 9.4% of all false accepts, and their mean IoU of 0.660 indicates moderate deviations that cover the target and merely include some extra background. The revision will add these numbers to Sec. 3.2 and to the limitations.
Weakness 2 (Insufficient analysis of computational efficiency and optimization stability). We added the average number of calls per stage (table below) and a stability analysis.
Average number of calls per stage:
About 67.3% (RefCOCO) to 77.6% (ReasonSeg) of the samples are accepted in the first round, and the cost is bounded by the three-round budget. For reference, the times in Table 4 are 15.3 s for ours, 15.2 s for Tarot-SAM3 and 23.2 s for EVOL-SAM3, but the three were measured under different protocols and hardware and should not be read as a controlled comparison.
On stability, the table below gives per-round gIoU (Qwen3-VL-4B, val splits). Every round brings a positive gain, with no oscillation of dropping and then recovering:
Among the samples that enter refinement (i.e., those judged Refine by the verifier at Round 0 and actually repaired), the per-sample stability statistics are:
Among the samples that enter refinement, about one-third (35.7%) gain substantially from refinement, about half (50.2%) remain essentially unchanged, and about 14.1% end up with a final mask worse than at Round 0. Samples accepted in the first round keep their Round 0 mask and fall outside the scope of this table, so over all samples the degradation rate is clearly lower than 14.1%. Overall, the net effect of refinement is positive, while a small fraction of samples degrade; this is a known limitation that we will report plainly in the revision, noting that the early-stopping mechanism reduces but cannot fully eliminate such cases. From the stopping distribution, about 67.3% (RefCOCO) to 77.6% (ReasonSeg) of the samples are accepted in the first round and stop early without consuming further budget; the fraction still not converged when the budget is exhausted is very low (<1%). Across three random seeds, the gIoU standard deviation of the full system on RefCOCO, RefCOCO+ and RefCOCOg val is 0.7–1.1 points, indicating good run-to-run stability.