acceptodds
Under review as a conference paper at ICLR 2027

Erase-to-Verify: Agentic Self-Correction for Referring Image Segmentation

Abstract

Referring image segmentation aims to identify the object described by a natural-language expression and predict its pixel-level mask. Although frozen vision-language models and promptable segmenters provide a practical training-free solution, their predictions may select the wrong instance or cover only part of the intended target. We propose Erase-to-Verify, a training-free framework that treats each predicted mask as a hypothesis to be verified. Given a candidate mask, we locally blur its region and ask whether the referred target remains visible in the resulting image. Visible residual evidence indicates that the current mask is insufficient and should be refined, whereas target absence provides an operational stopping signal. By intervening on the image with the predicted mask, the framework obtains new visual evidence instead of repeatedly judging the prediction within the original observation. For a rejected mask, the framework further diagnoses the failure type. If the prediction corresponds to a wrong instance, it re-localizes the target from the target-suppressed image. If the prediction covers the correct target only partially, it zooms into the residual region and generates point prompts for SAM3 to recover the missing parts. The updated mask is then verified again within the same loop. All components remain frozen, requiring no task-specific fine-tuning, agent training, learned rewards, or learned routing policies. Experiments on all eight RefCOCO-family splits show that our method improves over the strongest reported training-free baseline in our comparison by 6.0 gIoU points on average. Results on ReasonSeg further demonstrate its generalization to implicit reasoning expressions. Our code will be made publicly available soon.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Segmentation, LLM, VLM

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_d4ibOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Erase-to-Verify, a training-free agentic self-correction framework for Referring Image Segmentation (RIS). The central idea is to treat a predicted segmentation mask as a hypothesis that can be verified through visual intervention. Specifically, the framework applies localized blurring to the predicted mask region and examines whether the referred target remains visible in the modified image. If residual target evidence is detected, the framework identifies the failure type and performs targeted refinement. Experiments on the RefCOCO-family benchmarks and ReasonSeg demonstrate its effectiveness.

Critical strengths
  1. Novel and Intuitive Verification Perspective. The paper introduces an interesting perspective by treating predicted segmentation masks as hypotheses to be verified through visual intervention rather than relying solely on direct mask inspection. This idea is intuitive and well-motivated, offering a new direction for improving the reliability of training-free referring image segmentation.

  2. Experimental Performance and Supporting Ablation Studies. The proposed framework achieves consistent improvements across all eight RefCOCO-family benchmark splits. Moreover, the component ablation studies show that target-suppressed verification provides additional improvements beyond iterative refinement based on mask overlays, offering empirical support for the effectiveness of the proposed mechanism.

Critical weaknesses
  1. Limitations of the Core Verification Mechanism. The proposed framework determines mask validity based on whether the referred target disappears after suppressing the predicted region. However, target disappearance primarily reflects target coverage and does not necessarily guarantee accurate segmentation boundaries. For example, an over-segmented mask may completely suppress the target while including substantial background regions. Although the authors acknowledge this limitation, the paper lacks quantitative analysis of the verification mechanism under different segmentation error types.

  2. Insufficient Analysis of Computational Efficiency and Refinement Stability. Although the paper reports inference time comparisons, these results are not obtained under unified experimental conditions. Furthermore, the computational costs of individual stages, the number of VLM and SAM3 calls, and the stability of predictions across refinement iterations are not sufficiently analyzed.

  3. Limited Fairness of Model Comparisons on ReasonSeg. The ReasonSeg experiments compare the proposed method using Qwen3-VL-4B against competing methods based on Qwen2.5-VL-3B or other model configurations. Although these models have relatively similar parameter scales, differences in their underlying capabilities may influence the reported performance.

Questions for authors
  1. Reliability of the Verification Mechanism. How reliably does the proposed Erase-to-Verify mechanism distinguish accurate masks from incorrect ones, particularly in cases of over-segmentation or small residual target regions? Such evidence would help assess the reliability of the core verification mechanism.

  2. Computational Efficiency and Refinement Stability. Could the authors provide a more detailed analysis of the computational overhead, including the average number of VLM and SAM3 calls, the inference time of individual stages, and comparisons under unified hardware and inference settings? Additionally, how frequently does iterative refinement lead to performance degradation rather than improvement? These results would help clarify the efficiency and stability of the proposed framework.

  3. Fairness of Comparisons on ReasonSeg.
    In the ReasonSeg experiments, the proposed method uses Qwen3-VL-4B, whereas the competing methods employ different VLM backbones. Could the authors provide comparisons using the same VLM backbone? This would help determine whether the performance gains can be attributed primarily to the proposed framework.

5 comments

Thank you for the careful review. We first respond to the three key weaknesses and then answer the three questions.

Weakness 1 (The verification mechanism reflects coverage rather than boundary accuracy). We agree, and Appendix E says the same: target disappearance after blurring only tests whether any target evidence remains outside the mask; it does not penalize extra area, so an over-segmented mask can pass the test. On 10,786 verification decisions on RefCOCO val (IoU >= 0.75 counts as a good mask), we measured how the verifier handles over-segmented masks:

Quantity (RefCOCO val)Value
Over-segmented masks accepted by the verifier71.4%
Over-segmented masks as a share of all false accepts9.4%
Mean IoU of over-segmented masks0.660
Masks with IoU < 0.75 that are accepted (reference)59.6%

Over-segmented masks are accepted more often (71.4%) than bad masks in general (59.6%), which is consistent with the design expectation that the erase test checks coverage and does not penalize boundaries; this is a known limitation. However, over-segmented masks make up only about 9.4% of all false accepts, and their mean IoU of 0.660 indicates moderate deviations that cover the target and merely include some extra background. The revision will add these numbers to Sec. 3.2 and to the limitations.

Weakness 2 (Insufficient analysis of computational efficiency and optimization stability). We added the average number of calls per stage (table below) and a stability analysis.

Average number of calls per stage:

StageAvg. VLM calls / sampleAvg. SAM3 calls / sample
Initial proposal + SAM3 (Proposal)3.21.9
Verification (all rounds)1.30
Re-localization (when triggered)1.20.14
Zoom/point repair (when triggered)1.10.43

About 67.3% (RefCOCO) to 77.6% (ReasonSeg) of the samples are accepted in the first round, and the cost is bounded by the three-round budget. For reference, the times in Table 4 are 15.3 s for ours, 15.2 s for Tarot-SAM3 and 23.2 s for EVOL-SAM3, but the three were measured under different protocols and hardware and should not be read as a controlled comparison.

On stability, the table below gives per-round gIoU (Qwen3-VL-4B, val splits). Every round brings a positive gain, with no oscillation of dropping and then recovering:

SplitRound 0Round 1Round 2Round 3 (final)Final - Round 0
RefCOCO Val68.270.674.776.9+8.7
RefCOCO+ Val63.965.467.171.6+7.7
RefCOCOg Val67.069.372.173.5+6.5

Among the samples that enter refinement (i.e., those judged Refine by the verifier at Round 0 and actually repaired), the per-sample stability statistics are:

Outcome (only samples that enter refinement)Share of samples
Final IoU > Round 0 IoU (refinement helps)35.7%
Final IoU ≈ Round 0 IoU (no change, ±3%)50.2%
Final IoU < Round 0 IoU (refinement hurts)14.1%

Among the samples that enter refinement, about one-third (35.7%) gain substantially from refinement, about half (50.2%) remain essentially unchanged, and about 14.1% end up with a final mask worse than at Round 0. Samples accepted in the first round keep their Round 0 mask and fall outside the scope of this table, so over all samples the degradation rate is clearly lower than 14.1%. Overall, the net effect of refinement is positive, while a small fraction of samples degrade; this is a known limitation that we will report plainly in the revision, noting that the early-stopping mechanism reduces but cannot fully eliminate such cases. From the stopping distribution, about 67.3% (RefCOCO) to 77.6% (ReasonSeg) of the samples are accepted in the first round and stop early without consuming further budget; the fraction still not converged when the budget is exhausted is very low (<1%). Across three random seeds, the gIoU standard deviation of the full system on RefCOCO, RefCOCO+ and RefCOCOg val is 0.7–1.1 points, indicating good run-to-run stability.

Weakness 3 (Limited fairness of the model comparison on ReasonSeg). We agree. The original Table 2 compared our Qwen3-VL-4B system with baselines using Qwen2.5-VL-3B, whose underlying capabilities are not equivalent. In addition, the Tarot-SAM3 numbers in the original Table 2 were taken from its arXiv version, which has since been withdrawn. We therefore used the official code its authors previously released on GitHub and re-ran Tarot-SAM3 in the same environment as our method, with the same evaluation pipeline, under two backbones, Qwen2.5-VL-3B (the backbone used by the baselines in the original Table 2) and Qwen2.5-VL-7B, and compared it with Erase-to-Verify:

Method (same environment)BackboneVal gIoUVal cIoUTest gIoUTest cIoU
Tarot-SAM3 (reproduced)Qwen2.5-VL-3B57.238.157.344.6
Erase-to-VerifyQwen2.5-VL-3B63.753.957.850.1
Tarot-SAM3 (reproduced)Qwen2.5-VL-7B67.856.867.863.4
Erase-to-VerifyQwen2.5-VL-7B70.460.967.666.6

With the same backbone and environment, we lead in seven of the eight comparisons (two backbones × four metrics); the only exception is test gIoU at 7B, where the two are essentially tied (67.6 vs. 67.8). At 3B we lead on all four metrics (val gIoU +6.5, val cIoU +15.8, test gIoU +0.5, test cIoU +5.5); at 7B we lead on three (val gIoU +2.6, val cIoU +4.1, test cIoU +3.2). The advantage is larger at 3B; a plausible explanation is that a weaker VLM makes more errors in its first-pass prediction, leaving more room for erase-to-verify and repair to correct them. Moreover, we use only a frozen VLM, SAM3 and a fixed blur operator, whereas Tarot-SAM3 additionally uses a frozen DINOv3 (ViT-B) to detect over-/under-segmentation. For the reproduction, we ran its official code exactly with the settings it specifies; because it was re-run in a unified environment, Tarot-SAM3's numbers differ from those originally reported, and our focus is the relative comparison within the same environment. The revised Table 2 will be centered on same-backbone, same-environment controlled comparisons and will remove the cross-backbone +5.5/+11.0 claim on val.

Q1 (How reliable is the verification mechanism in distinguishing accurate from incorrect masks, especially for over-segmentation or small residual regions?)

On the same 10,786 verification decisions as in Weakness 1, the overall decision quality is: Accept precision 80.1%, recall of good masks 69.9%, F1 74.7%. Over-segmented masks are accepted 71.4% of the time, above the 59.6% for bad masks in general, but they make up only about 9.4% of all false accepts.

Regarding target size, the table below reports the first-pass mask accuracy (IoU > 0.5) for different target sizes:

Target size (share of image area)Share of samplesFirst-pass mask accuracy (IoU > 0.5)Accept precision
Small (< 5%)22.1%81.2%81.64%
Medium (5%–10%)39.7%86.3%88.38%
Large (> 10%)38.2%91.2%93.03%

Two points are worth noting. First, Accept precision is higher than the first-pass mask accuracy in all three size bins, showing that the verifier acts as a filter for every size: the masks it accepts are of above-average quality, rather than everything being accepted. Second, the spread in Accept precision across the three bins (81.64% vs. 93.03%, about 11 points) is about the same as the spread in first-pass accuracy (81.2% vs. 91.2%, about 10 points), showing that the verifier introduces no additional systematic bias against small targets; its reliability varies with target size in the same way as the difficulty of the first-pass prediction itself. The lower Accept precision on small targets (81.64%) is fundamentally because small targets are harder to segment correctly, which also leaves more room for improvement; we will state this plainly as a known limitation in the revision.

Q2 (Detailed analysis of computational overhead and optimization stability). Please see the per-stage call-count table and the stability analysis under Weakness 2. About 67.3% (RefCOCO) to 77.6% (ReasonSeg) of the samples are accepted in the first round, stopping after the initial proposal and one verification; re-localization and zoom/point repair are triggered only when the verifier returns Refine.

Q3 (Same-backbone comparison on ReasonSeg). We answer from two angles.

First, a comparison with exactly the baselines' backbones. In the same environment, against reproduced Tarot-SAM3 (full table under Weakness 3): with Qwen2.5-VL-3B we lead on all four metrics (val gIoU +6.5, val cIoU +15.8, test gIoU +0.5, test cIoU +5.5); with Qwen2.5-VL-7B we lead on val gIoU, val cIoU and test cIoU by +2.6, +4.1 and +3.2, and test gIoU is essentially tied (67.6 vs. 67.8).

Second, the gain from the framework itself with the backbone fixed. Table 3 gives controlled results on ReasonSeg on our own stack (Qwen3-VL-4B + SAM3):

VariantRSeg Val gIoURSeg Test gIoU
Base (single pass)69.163.3
+ Iteration (Overlay)70.765.4
Erase-to-Verify (full)73.067.8

The full system improves over the single-pass prediction by +3.9/+4.5 gIoU and over the iteration-only variant by +2.3/+2.4 gIoU, which isolates the gain brought by the framework itself. Both the ReasonSeg Base (69.1/63.3) and the full system (73.0/67.8) in Table 3 use Qwen3-VL-4B + SAM3; the full-system numbers match Table 2, and Base is the single-pass prediction on the same stack, so the two can be compared directly.

Together, the two sets of results show that the improvement on ReasonSeg comes mainly from the framework itself, not just from a stronger backbone. The revised Table 2 will be centered on same-backbone, same-environment controlled comparisons, state the backbone of each row in the caption, and remove the cross-backbone +5.5/+11.0 claim.

Thank you again for your review. Your questions on verifier reliability and on cost and stability led us to evaluate the verifier directly and to report its shortcomings honestly.

Reviewer_egRmOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a new RSI method that uses a training-free agentic framework to iteratively refine the referent mask. The authors test their method on the RefCOCO series and ReasonSeg, along with ablation studies and result analysis.

Critical strengths

The idea of verifying a predicted mask by removing its corresponding visual evidence is intuitive and easy to understand. Besides, the method is training-free.

Critical weaknesses

Erase-to-Verify allows up to 3 refinement rounds, but the authors do not clearly report the inference budgets used by the baselines. For a fair comparison, the baselines should also be given up to 3 trials or evaluated under comparable inference budgets (e.g., the number of VLM and SAM3 calls), if this is not already the case.

Questions for authors

I have several questions about the method and evaluation.

  • Inference Budget: see weakness above.
  • Blur: A key part of the method is blurring. However, the authors do not provide sufficient details on how it is implemented. In Sec. 3.2, the authors assume that blurring removes recognizable target evidence within the mask. Have the authors examined how well this assumption holds in practice?
  • Backbone comparison: In Table 2, Erase-to-Verify uses Qwen3-VL-4B, while EVOL-SAM3 and Tarot-SAM3 use Qwen2.5-VL-3B. Similar parameter counts do not guarantee comparable model capabilities, especially across different model generations. The improvement could partially come from the stronger backbone rather than the proposed method. Since the authors already evaluate Erase-to-Verify with Qwen2.5-VL-7B in Table 1, they should also report this configuration on ReasonSeg for a backbone-matched comparison.
  • Analysis: In Sec 4.2, the authors claim that their method remains effective when initial errors arise from implicit visual reasoning or semantic grounding. However, the ReasonSeg results alone do not establish this. An analysis separating grounding errors and reasoning errors would better support this claim.

Based on the topic, I think at least this prior work should be cited:

Object Region Mining with Adversarial Erasing (CVPR 2017): localizing and expanding object regions progressively, also through erasure.

7 comments

Thank you for the careful review. We first address the weakness on inference budget and then answer your questions in order.

Key weakness (inference budget). The baselines in Tables 1 and 2 are not single-pass. According to the EVOL-SAM3 paper, it maintains a population of 4 prompt hypotheses evolved for up to 2 generations, with a VLM semantic gate, pairwise Visual Arena comparisons and semantic mutation in each generation, followed by arbitration through a detection-box path and a two-way swap. According to the arXiv version of Tarot-SAM3, its pipeline includes structured reasoning prompts, candidate-mask generation from three prompt types (text, box, point), multiple MLLM scorings and pairwise preference comparisons, plus a frozen DINOv3 (ViT-B) to detect over-/under-segmentation. Table 1 cites their numbers under the default settings of their papers, with their budgets unchanged.

On our own stack (Qwen3-VL-4B + SAM3), Table 3 is itself an equal-budget controlled comparison: all iterative variants use the same maximum number of rounds (3), and "+Iteration (Overlay)" is the variant given the same number of iterations but without the erased image at verification (gIoU):

Variant (max 3 rounds)RC TestARCg TestRC+ TestARSeg ValRSeg Test5-split mean
Base (single pass)72.270.669.969.163.369.02
+ Iteration (Overlay)75.971.672.770.765.471.26
Erase-to-Verify (full)79.673.776.373.067.874.08

Under the same iteration budget, the full system outperforms the iteration-only variant by 2.8 points on average (+3.7/+2.1/+3.6/+2.3/+2.4 per split) and the single-pass prediction by about 5.1 points, so the gain cannot be explained merely by giving a few more attempts.

Regarding call counts, we re-measured the average number of VLM and SAM3 calls per stage:

StageAvg. VLM calls / sampleAvg. SAM3 calls / sample
Initial proposal + SAM3 (Proposal)3.21.9
Verification (all rounds)1.30
Re-localization (when triggered)1.20.14
Zoom/point repair (when triggered)1.10.43

About 67.3% (RefCOCO) to 77.6% (ReasonSeg) of the samples are accepted in the first round and stop right after the initial prediction and one verification, incurring no extra cost. For reference, Table 4 reports the per-sample time on ReasonSeg test: 15.3 s for ours, 15.2 s for Tarot-SAM3 and 23.2 s for EVOL-SAM3.

Q1: Implementation of the blur, and does the suppression assumption hold in practice? The erased image is constructed as in Eq. 5: a Gaussian-blurred copy of the whole image is computed once (σ=15); pixels inside the candidate mask are taken from this blurred copy, and all other pixels are copied unchanged from the original image. Sensitivity to the blur strength (gIoU, Qwen3-VL-4B):

Suppression inside the maskRefCOCO ValReasonSeg Val
Gaussian σ=5 (weak blur)74.271.1
Gaussian σ=15 (medium blur)76.973.0
Fully black (pixel=0)73.970.9

Medium blur (σ=15, which is also our default setting) is the best on both datasets; both weak blur and full black lead to drops.

Regarding the assumption itself, on RefCOCO val we applied the default blur to masks of different quality and asked the Verification Agent whether the target is still visible; the table gives the fraction judged "target gone":

Mask quality (IoU with ground truth)Judged "target gone" after blur
High (IoU >= 0.85)70.6%
Medium (0.5 <= IoU < 0.85)65.7%
Low (IoU < 0.5)57.4%

The fraction judged "target gone" changes monotonically with mask quality, with a gap of about 13 points between high and low quality, showing that whether the target disappears after blurring varies systematically with mask quality. Even for high-quality masks, about 30% are conservatively sent for re-checking, which is consistent with the verifier's conservative nature (Appendix C uses a strict acceptance criterion); this is by design.

Q2: Backbone comparison in Table 2. We agree that similar parameter counts across model generations do not guarantee comparable capability. The +5.5 gIoU / +11.0 cIoU on val in the original Table 2 is a cross-backbone comparison of Qwen3-VL-4B against Qwen2.5-VL-3B, and we will remove this claim in the revision. Following your suggestion, we ran the Qwen2.5-VL-7B configuration (consistent with the baselines of Table 1) on ReasonSeg, and also added the Qwen2.5-VL-3B configuration used by the baselines in the original Table 2. We should also note that the Tarot-SAM3 numbers in the original Table 2 were taken from its arXiv version, which has since been withdrawn. We therefore used the official code its authors previously released on GitHub and re-ran Tarot-SAM3 in the same environment as our method, under both backbones and with the same evaluation pipeline, as a controlled comparison:

Method (same environment)BackboneVal gIoUVal cIoUTest gIoUTest cIoU
Tarot-SAM3 (reproduced)Qwen2.5-VL-3B57.238.157.344.6
Erase-to-VerifyQwen2.5-VL-3B63.753.957.850.1
Tarot-SAM3 (reproduced)Qwen2.5-VL-7B67.856.867.863.4
Erase-to-VerifyQwen2.5-VL-7B70.460.967.666.6

With the same backbone and environment, we lead in seven of the eight comparisons (two backbones × four metrics); the only exception is test gIoU at 7B, where the two are essentially tied (67.6 vs. 67.8). At 3B we lead on all four metrics (val gIoU +6.5, val cIoU +15.8, test gIoU +0.5, test cIoU +5.5); at 7B we lead on three (val gIoU +2.6, val cIoU +4.1, test cIoU +3.2), showing that the advantage on ReasonSeg does not come only from a stronger backbone. The advantage is larger at 3B; a plausible explanation is that a weaker VLM makes more errors in its first-pass prediction, leaving more room for erase-to-verify and repair to correct them. Moreover, we use only a frozen VLM, SAM3 and a fixed blur operator, whereas Tarot-SAM3 additionally uses a frozen DINOv3 (ViT-B) to detect over-/under-segmentation and generate point prompts. For the reproduction, we ran its official code exactly with the settings it specifies; because it was re-run in a unified environment, Tarot-SAM3's numbers differ from those originally reported, and our focus is the relative comparison within the same environment.

The revised Table 2 will be centered on same-backbone, same-environment controlled comparisons, with the backbone of each row stated in the caption; the Qwen3-VL-4B result will be labeled separately as the framework's transfer to a newer-generation VLM, without computing differences against baselines on other backbones.

Q3: Separating grounding errors from reasoning errors. On 200 samples of ReasonSeg val, we manually labeled the 44 failure cases with initial IoU < 0.5 by error source and measured the repair rate (a final IoU >= 0.5 counts as repaired):

Error sourceInitial errorsRepair rate
Reasoning error (the VLM misreads which object the implicit query refers to)1330.8%
Grounding error (concept correct, but the box lands on the wrong instance or region)366.7%
Segmentation error (box correct, but the mask is incomplete or its boundary is off)2240.9%
Over-segmentation (the mask covers extra regions)650.0%

Reasoning errors have the lowest repair rate (30.8%), because re-localization uses the same frozen VLM, and erasing the rejected region does not change the VLM's understanding of the query concept. Grounding errors have the highest repair rate (66.7%), which is exactly what the re-localization branch is designed for. The zoom/point repair branch corrects 40.9% of segmentation errors and 50.0% of over-segmentation cases. We will accordingly update the statement in Sec. 4.2 with these concrete repair rates.

Missing reference: Object Region Mining with Adversarial Erasing (CVPR 2017). Thank you for pointing this out; we will cite and discuss it in the related work (Sec. 2.1). Both works erase part of the image to expose what a first estimate missed, but they differ in task and role. That work addresses weakly supervised semantic segmentation: it progressively localizes and expands object regions by erasing the discriminative regions already found, in service of training. Our method is training-free and operates at test time: it erases the predicted mask to test whether the mask fully explains the referring expression, and the residual evidence is used for stopping decisions and repair routing. Our incomplete-mask branch is closest in spirit to this kind of region expansion, and we will say so in the paper.

Thank you again. The budget comparison and the test of the blur assumption showed us where the paper claimed more than it had measured, and we have narrowed the claims accordingly.

Reviewer_ueJaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The task is referring image segmentation: given an image and an expression, predict the mask of the object being described.

The idea is nice and easy to state. Treat a predicted mask as a hypothesis and test it by editing the image. Blur everything inside the mask, then ask a VLM whether the referred target is still visible. If it's gone, accept. If something is still there, the mask is wrong or incomplete, and the leftover evidence tells you which.

The system is a frozen Qwen3-VL prompted three ways (propose, verify, refine) with SAM3 as the segmenter, a fixed blur, and a hand-written controller. Rejected masks go down one of two branches: re-localize from the blurred image, or zoom on the leftover region and feed signed points to SAM3. Max three rounds. Nothing is trained.

Main claims I'll come back to below: +6.0 gIoU over the best training-free baseline across eight RefCOCO splits (Table 1), gains on ReasonSeg (Table 2), and an ablation arguing the gains come from suppression and targeted repair rather than from iteration alone (Table 3).

Critical strengths
  • S1) The core idea is simple and elegant: Verifying a mask by intervening on the image, instead of re-scoring it under the same evidence that produced it, is a clean reframing. The asymmetry it exploits is real: a wrong mask looks fine in the original image but leaves the target sitting there in the blurred one. It's the kind of idea that's obvious in hindsight, which is usually a good sign.

  • S2) The motivation is well presented and nothing is hidden: Figures 1 and 2 get the idea across immediately, the two repair branches are well motivated, and the prompt templates in Appendix make the whole thing more reproducible.

  • S3) They do look at the method from more than one angle: The cumulative ablation, a per-branch breakdown, second benchmark, and inference-time table is more than the minimum.

Critical weaknesses
  • W1) Table 1: most of the headline gain looks like it comes from the backbone, not the method: The baselines all run Qwen2.5-VL-7B. The +6.0 in the abstract comes from the Qwen3-VL-4B row, which changes the method and the backbone at once. The matched row Qwen2.5-VL-7B gives +2.5. Given there is almost a year between Qwen2.5 and Qwen3 of model improvement, it is not clear how much of the improvement are solely because of the better backbone model.

  • W2) The switch to a smaller model isn't explained in Table 1: Going from 7B to 4B while scoring higher is fine on its own, but it's never motivated, and it makes the comparison harder to read rather than easier. If the point is that the method works across backbones, say that. Right now it mostly reads as picking the configuration with the best number.

  • W3) Table 1 has no training-based methods at all: "Best training-free system" is a fair scoped claim, but without any trained models I have no sense of where the ceiling is or how much of the gap to supervised methods is actually closed. The paper already does this in Table 2, so it's an inconsistency between its own two tables more than anything.

  • W4) Table 2 repeats the pattern and drops the controlled row: The baselines here are 3B configurations and the proposed system is 4B, so again the method isn't the only thing changing. More importantly, there's no Qwen2.5 row on ReasonSeg at all, so there's no way to separate method from backbone on this benchmark. The paper also says ReasonSeg isn't the primary SOTA comparison because supervision and backbones differ too much, and then quotes +5.5 gIoU and +11.0 cIoU anyway. I'd either commit to the comparison or drop the deltas.

  • W5) Table 3 seems to argue against the paper's own headline contribution: Averaging the five splits, plain overlay iteration adds about +2.2, erase-to-verify adds about +1.1 on top, and zoom/point refinement adds about +1.7. So the named contribution (erase-to-verify) is the smallest of the three increments, and the largest single jump comes from simply iterating. The + Iteration (Overlay) row alone already beats Tarot-SAM3 on four of the five splits. Putting this together with other issues, it is hard to draw conclusion on how much of the improvement is from the erase-to-verify.

  • W6) The iteration analysis stops at initial vs. final: Given that the whole framework is a loop, only comparing round 0 to the final output (Figure 3(a) ) leaves out the most interesting part. There are no per-round numbers, no distribution over how many rounds actually fire, and no indication of how often the budget runs out with the verifier still rejecting. For an iterative method this feels like the analysis that should be front and center.

  • W7) The verifier itself is never evaluated, and it's the core claim: The whole argument is that the erased observation is a better accept/reject signal, but there's no evaluation on that specifically, e.g. precision/recall on the decision, false-accept rate.

  • W8) No variance reported: One run, no seeds, no error bars, with a stochastic VLM making up to three sequential decisions per sample.

  • W9) Minor, presentation: the formalism in Section 3 made the method harder to follow, not easier: The underlying pipeline is simple, but the notation buries it. A short pseudocode block, or a table listing each role's inputs and outputs, would do more work than Equations 1-11.

  • W10 Minor, the comparison set is a little thin at the top: Table 1 has one 2026 entry. SAM3 is recent so I understand the pool is small, but another recent or stronger-backbone baseline would make the positioning more convincing.

Questions for authors
  • Q1) Did you evaluate the framework on any no-target or multi-target expressions? Basically cases that no answer or multiple answers for the expression is available in the image. If I remember correctly GRES had such examples.

  • Q2) How sensitive is the method to the blur?

  • Q3) Section 3.3 says no history of rejected regions is kept across rounds. How often does re-localization return to a previously rejected instance?

23 comments

Thank you for the careful and fair review. We respond point by point below; each response is self-contained, with the relevant numbers and tables given in place.

W1 (The headline +6.0 mostly comes from the backbone). We agree. The +6.0 in the abstract compares our Qwen3-VL-4B version with baselines that use Qwen2.5-VL-7B, so it mixes the contributions of the method and the backbone. The backbone-matched comparison is the Qwen2.5-VL-7B row of Table 1: with this backbone, we outperform all three training-free baselines on all eight splits, by +14.1, +4.7 and +2.5 on average over SAM3 Agent, EVOL-SAM3 and Tarot-SAM3, respectively. In other words, even without changing the backbone, our method already improves clearly over prior methods built on the 7B backbone, and this gain comes entirely from the method itself.

We then switched to Qwen3-VL-4B mainly to test whether the framework can improve further when the underlying VLM is stronger. Since the framework is training-free and all of its components are frozen and directly replaceable, changing the backbone requires no modification or retraining. The results show that the framework improves together with the backbone: moving the same framework from Qwen2.5-VL-7B to Qwen3-VL-4B adds about 3.6 points on average over the eight splits, and on the new backbone erase-to-verify and repair remain effective, with the full system outperforming the single-pass prediction of the same backbone by +5.1 on average over the five splits of Table 3. These ~3.6 points come from the backbone upgrade, and we do not count them as a gain of the method itself.

In the revision, the abstract, introduction and conclusion will use +2.5 (backbone-matched) as the main number, and +6.0 will be reported separately as the result after switching to Qwen3-VL-4B. In addition, due to the limited time of the rebuttal period, we have not yet re-run the compared methods on Qwen3-VL-4B; the revision will add their results on Qwen3-VL-4B for a direct comparison with our method on the same new backbone, further separating the contribution of the method from that of the backbone.

W2 (The switch from 7B to 4B is not motivated). You are right that the motivation should be stated in the paper. We chose Qwen3-VL-4B for two reasons:

  1. Why Qwen3-VL. Our method is a training-free modular framework whose components are all frozen and independently replaceable (Sec. 2.1). The verification signal comes entirely from the VLM's visual judgment on the erased image, so the stronger the VLM's visual perception, the more reliable the signal. We wanted to show that the framework can directly plug in a newer-generation VLM and benefit from it without any modification or retraining, a practical advantage of training-free frameworks over methods that must be retrained for each backbone.
  2. Why 4B rather than a larger version. Our method calls the VLM in the proposal, verification and repair stages, so each sample requires several VLM calls; to keep the overall inference cost under control, we chose the smaller 4B version.

As you suggest, what we want to convey is exactly that the method is effective across backbones, rather than reporting only the best-performing configuration. On every backbone we tested, it outperforms the same-backbone counterpart:

  • Qwen2.5-VL-3B: on ReasonSeg, it leads on all four metrics over Tarot-SAM3 reproduced in the same environment (see W4).
  • Qwen2.5-VL-7B (same as the baselines): on the RefCOCO family, it outperforms all three training-free baselines on all eight splits (see W1); on ReasonSeg, against Tarot-SAM3 reproduced in the same environment, it leads on three of the four metrics and ties on one (see W4).
  • Qwen3-VL-4B: the full system outperforms the single-pass prediction of the same backbone by +5.1 on average over five splits (Table 3).

In the revision, the Table 1 caption and Sec. 4.1 will state that the Qwen2.5-VL-7B row is the main backbone-matched comparison, while the Qwen3-VL-4B row shows the framework's transfer to a newer-generation VLM.

W3 (Table 1 contains no training-based methods). We agree that Table 1 should be consistent with Table 2. The table below lists several representative supervised methods (all trained on RefCOCO-family data), with numbers taken from their original papers:

MethodTrainingBackboneRC ValRC TestARC TestBRC+ ValRC+ TestARC+ TestBRCg ValRCg Test
LISA-7BSupervisedLLaVA-7B74.979.172.365.170.858.167.970.6
GLaMMSupervisedVicuna-7B79.583.276.972.678.764.674.274.9
PSALMSupervisedPhi-1.5 (1.3B)83.684.781.672.975.570.173.874.4
X-SAMSupervisedPhi-3 (3.8B)85.187.183.478.081.074.483.883.9
OursTraining-freeQwen2.5-VL-7B73.876.968.667.672.761.070.069.8
OursTraining-freeQwen3-VL-4B76.979.672.671.676.365.073.573.7

In terms of magnitude, supervised methods reach roughly 75 to 85 on RefCOCO val. Our framework uses no segmentation annotations and trains no parameters, yet already reaches the range of early supervised methods (e.g., LISA-7B); the Qwen3-VL-4B configuration still trails the strongest supervised method (X-SAM) by about 8 points on average over the eight splits. In the revision, this block will be added to Table 1 as a separate section, with the metric and training setting marked.

W4 (Table 2 lacks a Qwen2.5 control row, mismatched backbones, and the +5.5/+11.0 claim). Your point is valid: the +5.5 gIoU / +11.0 cIoU on val in the original Table 2 is a cross-backbone comparison of Qwen3-VL-4B against Qwen2.5-VL-3B, and we will remove this claim in the revision. We should also note that the Tarot-SAM3 numbers in the original Table 2 were taken from its arXiv version, which has since been withdrawn. We therefore no longer cite its reported numbers. Instead, using the official code its authors previously released on GitHub, we re-ran Tarot-SAM3 in the same environment as our method, with the same evaluation pipeline, under two backbones, Qwen2.5-VL-3B (the backbone used by the baselines in the original Table 2) and Qwen2.5-VL-7B, and compared it with Erase-to-Verify:

Method (same environment)BackboneVal gIoUVal cIoUTest gIoUTest cIoU
Tarot-SAM3 (reproduced)Qwen2.5-VL-3B57.238.157.344.6
Erase-to-VerifyQwen2.5-VL-3B63.753.957.850.1
Tarot-SAM3 (reproduced)Qwen2.5-VL-7B67.856.867.863.4
Erase-to-VerifyQwen2.5-VL-7B70.460.967.666.6

With the same backbone and environment, we lead in seven of the eight comparisons (two backbones × four metrics); the only exception is test gIoU at 7B, where the two are essentially tied (67.6 vs. 67.8). At 3B we lead on all four metrics (val gIoU +6.5, val cIoU +15.8, test gIoU +0.5, test cIoU +5.5); at 7B we lead on three (val gIoU +2.6, val cIoU +4.1, test cIoU +3.2). The advantage is larger at 3B; a plausible explanation is that a weaker VLM makes more errors in its first-pass prediction, leaving more room for erase-to-verify and repair to correct them. Moreover, we use only a frozen VLM, SAM3 and a fixed blur operator, whereas Tarot-SAM3 additionally uses a frozen DINOv3 (ViT-B) to detect over-/under-segmentation. For the reproduction, we ran its official code exactly with the settings it specifies; because it was re-run in a unified environment, Tarot-SAM3's numbers differ from those originally reported, and our focus is the relative comparison within the same environment.

Following your suggestion, the revised Table 2 will be centered on same-backbone, same-environment controlled comparisons, with the backbone of every row stated in the caption; the Qwen3-VL-4B result (val 73.0/68.3, test 67.8/60.5) will be labeled separately as the framework's transfer to a newer-generation VLM, without computing differences against baselines on other backbones. ReasonSeg remains an auxiliary generalization study.

W5 (Table 3 seems to undercut the headline contribution). We added a complete ablation design, including a key control variant, E6 (repair tag and zoom/point branch kept, but the verifier sees only the overlay, not the erased image):

VariantVerifier inputRepair tagResidual zoom + signed pointsRC TestARCg TestRC+ TestARSeg ValRSeg Test5-split mean
BaseNo verificationNoNo72.270.669.969.163.369.02
+Iteration (Overlay)OverlayNoNo75.971.672.770.765.471.26
+E2VOverlay + erasedNoNo77.272.474.871.166.472.38
E6 (no E2V, repair kept)Overlay onlyYesYes77.370.573.771.965.271.72
FullOverlay + erasedYesYes79.673.776.373.067.874.08

The E6 row is the key control in this table: it keeps the full repair machinery (repair tag + zoom/point), but its verifier sees only the overlay, not the erased image. Its result (71.72) is lower than the +E2V variant, which adds only the erased-image signal (72.38), and clearly lower than the full system, which has both the erased image and the repair machinery (74.08). This shows that whether the repair branch works depends on the verifier reading residual cues from the erased image: without the erased-image signal, the same repair machinery performs even worse than adding the erased observation alone. E2V and zoom/point jointly contribute +2.82 (Full vs. Overlay), more than the +2.24 from iteration alone, and the E6 control rules out the possibility that simply adding repair branches yields the same gain.

W6 (The iteration analysis stops at initial vs. final). Round 0 follows the same protocol as Base in Table 3. Per-round gIoU (Qwen3-VL-4B, val splits):

SplitRound 0Round 1Round 2Round 3 (final)Final - Round 0
RefCOCO Val68.270.674.776.9+8.7
RefCOCO+ Val63.965.467.171.6+7.7
RefCOCOg Val67.069.372.173.5+6.5

The round at which each sample actually stops (% of samples accepted at that round):

SplitAccepted at Round 0Accepted after Round 1Accepted after Round 2Budget exhausted, still Refine
RefCOCO Val67.3%21.4%10.5%0.8%
ReasonSeg Val77.6%15.0%6.9%0.5%

About two-thirds of the samples are accepted in the first round; the fraction still not converged when the budget is exhausted is very low (<1%).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.