- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a dual-branch framework for referring image segmentation (RIS). The first branch directly feeds the text expression into SAM3 to obtain a segmentation mask. The second branch uses an object detector to construct a scene graph and parses the text into relational constraints. It then performs symbolic reasoning to locate the target and uses SAM3 to generate the corresponding segmentation mask. Finally, a gating mechanism (DHMA) selects between the predictions from the two branches to produce the final result.
Critical strengths
Strengths
-
The paper addresses an important problem in RIS, especially the difficulty of handling spatial relationships. Combining explicit scene graph reasoning with SAM3 is an interesting idea.
-
The dual-hypothesis design is reasonable. Instead of relying entirely on symbolic reasoning, the method uses SAM3 as the default and selectively switches to the symbolic prediction. The learned gates also show promising improvements across multiple datasets.
Critical weaknesses
Main Reasons for Rejection
-
Unclear zero-shot setting. Lock C and Lock D are trained on RefCOCOg validation data, with IoU-based performance used for model selection. If ground-truth RIS masks are involved, I don't think it is fair to compare them directly with fully zero-shot methods. This needs to be clarified.
-
Concerns about comparison fairness. The SAM3-text baseline is described as using the head noun phrase, while ReSCU uses the full expression. Also, ReSCU benefits from the much stronger SAM3 backbone, but there is no clear ablation showing how much improvement comes from the backbone itself.
-
Insufficient evidence for the main contribution. The paper emphasizes relational reasoning, but most of the improvement seems to come from the learned DHMA gate. A graph-free baseline or scene graph ablation is needed to show whether the proposed relational reasoning is actually necessary.
Questions for authors
Weaknesses
-
Lock C and Lock D are both trained with a classification head on the RefCOCOg validation set, and IoU-based performance is used for hyperparameter tuning. Does this really fit the definition of zero-shot learning? Lock A brings almost no improvement, while the gains mainly appear after fitting the learned gates. If the gate relies on some ground-truth RIS masks to make its decisions, I think this method should be considered weakly supervised or partially supervised rather than zero-shot.
-
The baseline definition needs to be clarified. In Section 3.6, the paper states that the appearance branch of ReSCU directly feeds the full expression into SAM3. However, in Section 4.1 and Appendix F, the matched SAM3-text baseline is described as using only the head noun phrase of the expression (is this the
nsubjextracted after parsing?). I am a bit confused about whether these two branches actually use the same text input. -
From the title, abstract, and introduction (especially the first contribution), the paper seems to emphasize relational reasoning as the main way to improve RIS performance. However, looking at the main experimental results, DHMA seems to be the more important contribution. I think an ablation study on the scene graph is missing here. For example, what would happen if the visual entities were not represented as a graph? How much would the performance change?
-
I also have some concerns about the fairness of the performance comparisons. As mentioned in Point 1, Lock C and Lock D are fitted on the validation set. Is it fair to compare their results directly with other methods under this setting? Wouldn't it be more reasonable to train the gate on the training set, use the validation set to select the best weights or hyperparameters, and then evaluate on the test set? Also, this paper uses SAM3, which is a much stronger segmentation backbone than those used in many previous methods. Is there any backbone ablation to show how much of the performance gain actually comes from SAM3 itself? I think this is quite important.
-
I would also suggest evaluating the method on more datasets or scenarios, such as ReferIt (which is somewhat similar to your Type S expressions). In addition, comparisons with some recent RIS methods are missing.
Additional Suggestions
These are more personal suggestions, but they also affected my score to some extent.
-
Figure 1 could be improved visually. I would suggest using vector graphics for the main pipeline figure. The arrangement of the images and other elements could also be refined to make the figure look cleaner and more polished.
-
I would suggest making the contributions more concise. There are quite a few of them right now, and I think focusing on 2–3 main contributions would be enough.
-
The main performance table feels a bit too limited. It would be better to include comparisons with other methods across RefCOCO, RefCOCO+, and RefCOCOg, rather than only showing the main results on RefCOCOg.
-
Some tables look a bit messy, with too much text mixed into the results (e.g., Table 1). I think the formatting could be cleaner.
-
Some fonts and formatting look a bit out of place, especially for terms like
HistGradientBoostingClassifier,min_promote_score, and"objects . items . things". It would be better to make the formatting more consistent. -
The writing could also be more concise in several places. For example, the abstract is quite long and could be shortened.
2 comments
I would like to thank the authors for their patient clarifications. However, considering the current state of the work in terms of writing, experimental results, and overall presentation, I have decided to maintain my original score.
Thank you for the detailed and constructive review. We respond to each main point.
1 (zero-shot). You are right. Gates are fit on RefCOCOg-val with labels from ground-truth mask IoU, so the system is weakly supervised and should not be compared as zero-shot. We will remove the claim and describe it as a training-free pipeline plus a small gate. As you suggest, we fit the gate on a RefCOCOg train subset (5,000 expressions, 4,616 images, none shared with val/test), chose only the threshold on val, and evaluated on test once: dense gate +2.95 [2.35, 3.56] (30.4% promoted), sparse gate +1.57 [1.09, 2.05]; frozen transfer (dense) to RefCOCO testA+B +1.56 [0.99, 2.15] and RefCOCO+ +1.64 [1.07, 2.23]. This matches or exceeds the val-fit gates (+2.44 dense / +1.59 sparse).
2 (baseline text). The "head noun phrase" description in Sec. 4.1 / App. F is wrong. All baseline and appearance-branch masks use the full expression (7,596/7,596 rows). Extra head-noun baseline (spaCy head NP, H100): val 50.57, test 52.43, vs. full 53.60 / 53.27. The gates beat both: on test, sparse/dense +2.44 [1.31, 3.54] and +3.28 [2.14, 4.42] over the head baseline.
3 (scene-graph ablation). We ran it. On RefCOCOg test (SAM3-text 53.27; paired CI): full dense gate +2.44 [1.79, 3.11]; relations removed from the symbolic branch, gate frozen +2.42 [1.77, 3.09]; relations removed, gate refit +2.72 [2.08, 3.39]; gate on SAM3-side features only +0.81 [0.52, 1.10], and with SAM3's confidence added +2.72 [2.25, 3.20]; random promotion at the same rate -4.16 ± 0.30. Your reading is right, and the data go further: explicit relations are not needed, and the gain is largely "switch to a detector-grounded mask when SAM3 is unsure." We will rescope title, abstract and contributions accordingly.
4 (fairness, backbone). The backbone-only comparison is SAM3-text with the full expression (53.27 on test): every delta in the paper is relative to it, so the reported gains are net of SAM3. We agree the CLIP-based baselines in Table 2 are not like-for-like and will say so. We will also follow your suggestion on the training protocol (see 1).
5 (datasets, recent methods). RefCOCO/RefCOCO+ results (frozen transfer of the RefCOCOg-selected gates) are in Table 3 of the submission; with the train-fit gate the dense gain is +1.56 [0.99, 2.15] (RefCOCO) and +1.64 [1.07, 2.23] (RefCOCO+). We did not run ReferIt or SAM 3 Agent; we will add ReferIt and a stronger training-free baseline in the revised version and report them either way. We agree the comparison set is limited and will say so.
Additional suggestions. We will use vector graphics for the pipeline figure, cut contributions to two or three, tidy Table 1 and code-font formatting, and shorten the abstract.