acceptodds
Under review as a conference paper at ICLR 2027

RESCU: RELATIONAL SCENE CONSTRAINT UNDERSTANDING FOR ZERO-SHOT REFERRING IMAGE SEGMENTATION

Abstract

Referring image segmentation asks a model to find the exact pixels an expression describes. Zero-shot methods almost universally reduce this to cross-modal retrieval: embed the expression and each candidate region in a shared CLIP space and take the best match. That formulation cannot represent relational constraintsleft_of, between, negation because no single embedding-space comparison ever sees the joint configuration of target and anchor. We introduce ReSCU (lational cene onstraint nderstanding), which builds an explicit relational scene graph over open-vocabulary proposals, parses the expression into a typed query, and evaluates it symbolically to produce a hypothesis - then arbitrates it against a strong hypothesis obtained by prompting a modern promptable-concept segmenter (SAM3) directly with the raw expression. The key idea, Dual-Hypothesis Mask Arbitration (DHMA), is to trust the appearance branch by default and promote the symbolic hypothesis only when a gate calibrated on held-out data says the parse is trustworthy. We study three such gates, of increasing capacity: hand-set thresholds, a sparse learned classifier, and a denser one over an expanded feature set. The hand-set gate () sits at statistical parity with matched SAM3-text and does not transfer across datasets. A sparse logistic gate over 17 features (), selected (grid fixed a priori, scored on held-out data, test touched once), produces a significant win on RefCOCOg (/ val/test mIoU, /pp, CI excluding zero) that transfers frozen to RefCOCO (/pp) and RefCOCO+ (/pp). Replacing lock C's classifier with a denser gradient-boosted one over 27 features () nearly doubles the margin (/ mIoU, /pp, CI test) and still transfers frozen to RefCOCO (/pp) and RefCOCO+ (/pp). Both gates lead every published zero-shot RIS number we could verify, including HybridGL (/) and RefChess (/), and both help all four expression types, not only the relational one lock A targeted. An oracle upper bound sits points above lock A's baseline; lock C closes of that gap and lock D , so roughly half remains unrealized under the best gate we found. Detection is not the bottleneck of gold targets survive to the proposal stagebut a frozen VLM used for spatial-edge prediction collapses to a single answer on held-out pairs, so our default edge backend stays geometric rather than learned. Lock D's larger gain also comes from intervening - more often through a harder-to-audit model, so we keep lock C as a sparse, interpretable alternative alongside lock D's headline result, and view closing the remaining oracle gap as the natural next step.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Referring image segmentation, Zero-shot learning, Neuro-symbolic reasoning, Relational scene graphs, Vision-language grounding

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_jdbtOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a dual-branch framework for referring image segmentation (RIS). The first branch directly feeds the text expression into SAM3 to obtain a segmentation mask. The second branch uses an object detector to construct a scene graph and parses the text into relational constraints. It then performs symbolic reasoning to locate the target and uses SAM3 to generate the corresponding segmentation mask. Finally, a gating mechanism (DHMA) selects between the predictions from the two branches to produce the final result.

Critical strengths

Strengths

  1. The paper addresses an important problem in RIS, especially the difficulty of handling spatial relationships. Combining explicit scene graph reasoning with SAM3 is an interesting idea.

  2. The dual-hypothesis design is reasonable. Instead of relying entirely on symbolic reasoning, the method uses SAM3 as the default and selectively switches to the symbolic prediction. The learned gates also show promising improvements across multiple datasets.

Critical weaknesses

Main Reasons for Rejection

  1. Unclear zero-shot setting. Lock C and Lock D are trained on RefCOCOg validation data, with IoU-based performance used for model selection. If ground-truth RIS masks are involved, I don't think it is fair to compare them directly with fully zero-shot methods. This needs to be clarified.

  2. Concerns about comparison fairness. The SAM3-text baseline is described as using the head noun phrase, while ReSCU uses the full expression. Also, ReSCU benefits from the much stronger SAM3 backbone, but there is no clear ablation showing how much improvement comes from the backbone itself.

  3. Insufficient evidence for the main contribution. The paper emphasizes relational reasoning, but most of the improvement seems to come from the learned DHMA gate. A graph-free baseline or scene graph ablation is needed to show whether the proposed relational reasoning is actually necessary.

Questions for authors

Weaknesses

  1. Lock C and Lock D are both trained with a classification head on the RefCOCOg validation set, and IoU-based performance is used for hyperparameter tuning. Does this really fit the definition of zero-shot learning? Lock A brings almost no improvement, while the gains mainly appear after fitting the learned gates. If the gate relies on some ground-truth RIS masks to make its decisions, I think this method should be considered weakly supervised or partially supervised rather than zero-shot.

  2. The baseline definition needs to be clarified. In Section 3.6, the paper states that the appearance branch of ReSCU directly feeds the full expression into SAM3. However, in Section 4.1 and Appendix F, the matched SAM3-text baseline is described as using only the head noun phrase of the expression (is this the nsubj extracted after parsing?). I am a bit confused about whether these two branches actually use the same text input.

  3. From the title, abstract, and introduction (especially the first contribution), the paper seems to emphasize relational reasoning as the main way to improve RIS performance. However, looking at the main experimental results, DHMA seems to be the more important contribution. I think an ablation study on the scene graph is missing here. For example, what would happen if the visual entities were not represented as a graph? How much would the performance change?

  4. I also have some concerns about the fairness of the performance comparisons. As mentioned in Point 1, Lock C and Lock D are fitted on the validation set. Is it fair to compare their results directly with other methods under this setting? Wouldn't it be more reasonable to train the gate on the training set, use the validation set to select the best weights or hyperparameters, and then evaluate on the test set? Also, this paper uses SAM3, which is a much stronger segmentation backbone than those used in many previous methods. Is there any backbone ablation to show how much of the performance gain actually comes from SAM3 itself? I think this is quite important.

  5. I would also suggest evaluating the method on more datasets or scenarios, such as ReferIt (which is somewhat similar to your Type S expressions). In addition, comparisons with some recent RIS methods are missing.

Additional Suggestions

These are more personal suggestions, but they also affected my score to some extent.

  1. Figure 1 could be improved visually. I would suggest using vector graphics for the main pipeline figure. The arrangement of the images and other elements could also be refined to make the figure look cleaner and more polished.

  2. I would suggest making the contributions more concise. There are quite a few of them right now, and I think focusing on 2–3 main contributions would be enough.

  3. The main performance table feels a bit too limited. It would be better to include comparisons with other methods across RefCOCO, RefCOCO+, and RefCOCOg, rather than only showing the main results on RefCOCOg.

  4. Some tables look a bit messy, with too much text mixed into the results (e.g., Table 1). I think the formatting could be cleaner.

  5. Some fonts and formatting look a bit out of place, especially for terms like HistGradientBoostingClassifier, min_promote_score, and "objects . items . things". It would be better to make the formatting more consistent.

  6. The writing could also be more concise in several places. For example, the abstract is quite long and could be shortened.

2 comments

Thank you for the detailed and constructive review. We respond to each main point.

1 (zero-shot). You are right. Gates are fit on RefCOCOg-val with labels from ground-truth mask IoU, so the system is weakly supervised and should not be compared as zero-shot. We will remove the claim and describe it as a training-free pipeline plus a small gate. As you suggest, we fit the gate on a RefCOCOg train subset (5,000 expressions, 4,616 images, none shared with val/test), chose only the threshold on val, and evaluated on test once: dense gate +2.95 [2.35, 3.56] (30.4% promoted), sparse gate +1.57 [1.09, 2.05]; frozen transfer (dense) to RefCOCO testA+B +1.56 [0.99, 2.15] and RefCOCO+ +1.64 [1.07, 2.23]. This matches or exceeds the val-fit gates (+2.44 dense / +1.59 sparse).

2 (baseline text). The "head noun phrase" description in Sec. 4.1 / App. F is wrong. All baseline and appearance-branch masks use the full expression (7,596/7,596 rows). Extra head-noun baseline (spaCy head NP, H100): val 50.57, test 52.43, vs. full 53.60 / 53.27. The gates beat both: on test, sparse/dense +2.44 [1.31, 3.54] and +3.28 [2.14, 4.42] over the head baseline.

3 (scene-graph ablation). We ran it. On RefCOCOg test (SAM3-text 53.27; paired CI): full dense gate +2.44 [1.79, 3.11]; relations removed from the symbolic branch, gate frozen +2.42 [1.77, 3.09]; relations removed, gate refit +2.72 [2.08, 3.39]; gate on SAM3-side features only +0.81 [0.52, 1.10], and with SAM3's confidence added +2.72 [2.25, 3.20]; random promotion at the same rate -4.16 ±\pm 0.30. Your reading is right, and the data go further: explicit relations are not needed, and the gain is largely "switch to a detector-grounded mask when SAM3 is unsure." We will rescope title, abstract and contributions accordingly.

4 (fairness, backbone). The backbone-only comparison is SAM3-text with the full expression (53.27 on test): every delta in the paper is relative to it, so the reported gains are net of SAM3. We agree the CLIP-based baselines in Table 2 are not like-for-like and will say so. We will also follow your suggestion on the training protocol (see 1).

5 (datasets, recent methods). RefCOCO/RefCOCO+ results (frozen transfer of the RefCOCOg-selected gates) are in Table 3 of the submission; with the train-fit gate the dense gain is +1.56 [0.99, 2.15] (RefCOCO) and +1.64 [1.07, 2.23] (RefCOCO+). We did not run ReferIt or SAM 3 Agent; we will add ReferIt and a stronger training-free baseline in the revised version and report them either way. We agree the comparison set is limited and will say so.

Additional suggestions. We will use vector graphics for the pipeline figure, cut contributions to two or three, tidy Table 1 and code-font formatting, and shorten the abstract.

I would like to thank the authors for their patient clarifications. However, considering the current state of the work in terms of writing, experimental results, and overall presentation, I have decided to maintain my original score.

Reviewer_AAAbOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents ReSCU, a zero-shot referring image segmentation method that integrates geometric scene graph construction with relation parsing. The core idea is a gating mechanism named DHMA, which dynamically selects between the symbolic hypothesis and the appearance hypothesis through different Lock protocols.

Critical strengths

The method supplements the CLIP-based retrieval with explicit relation reasoning, resulting in an intuitive computational process.

Critical weaknesses

1. Clarity issues.​ The abstract and introduction sections do not follow standard formatting conventions. More critically, the core content regarding Lock A/B/C/D appears scattered across different sections of the paper, making the narrative difficult to follow.

2. Limited methodological novelty.​ The key idea, DHMA, can essentially be reduced to manually set thresholds and simple classifiers. It remains unclear whether the principle or methodology constitutes a sufficient technical contribution.

Questions for authors
  1. In the comparison with other methods (Table 2), how much of the reported advantage is attributable to the default SAM3 branch rather than the proposed approach itself?
  2. The threshold parameters for Lock A are tuned on the source dataset. How well do these thresholds transfer to unseen datasets?
  3. Lock C and Lock D differ only in feature dimensionality and classifier choice. Is there any compelling reason to retain both, rather than simply adopting the strongest variant?
  4. Sections 3.6 and 3.7 introduce Lock A, Lock C, and Lock D out of order, with Lock B relegated to the appendix. Why was this organization chosen, given that it disrupts the logical flow and readability of the paper?
  5. Since the system defaults to the appearance hypothesis, does this imply that SAM3 alone suffices in most cases? How should we evaluate the complementary value and necessity of the symbolic hypothesis?
2 comments

Thank you. We address the weaknesses and each question.

W1 (clarity / lock organization). Fair. We will present the three gates once, as "threshold," "sparse," and "dense" gates, in a single section, move the lock-B history to the appendix, and fix the abstract/introduction structure.

W2 (novelty). We agree the mechanism is simple: a gate over two masks. We do not claim architectural novelty, and we no longer present DHMA as the contribution. We now think the paper's contribution is a diagnosis: when does a symbolic hypothesis add value over a strong promptable segmenter. New controls show the gain comes from a confidence-triggered fallback to a second, detector-grounded mask, not from relations (relations removed: +2.42 vs +2.44; SAM3-side gate with SAM3 confidence: +2.72 [2.25, 3.20] at 11.7% switching). This is a useful finding for anyone building structured pipelines on promptable segmenters, and we will frame the paper that way.

Q1 (how much of Table 2 is SAM3?). Most of the gap to the CLIP-based methods. SAM3-text alone scores 53.27 on RefCOCOg test, above every published zero-shot number in Table 2. The contribution of our method is the delta over SAM3-text, +1.59 [1.18, 2.02] (sparse) and +2.44 [1.79, 3.11] (dense) on test (GDINO dumps). We will rescope Table 2 accordingly.

Q2 (lock A transfer). It does not transfer: on RefCOCO/RefCOCO+ val/test, lock A's deltas are -0.089, -0.310, -0.110, -0.214 (all CIs include 0; Table 3 of the submission). We report this as a negative result.

Q3 (keep both C and D?). The sparse gate promotes about 7% of images and is easy to audit; the dense gate promotes about 32% and gives a larger gain (+1.59 vs. +2.44 on test). We will present the dense gate as the main result and the sparse gate as an interpretable variant in one place.

Q4 (organization). Agreed (W1).

Q5 (does defaulting to the appearance branch mean SAM3 suffices?). Mostly yes, and new controls sharpen this. The oracle prefers the symbolic mask on 37.9% of test images (+11.43 [10.71, 12.17]), but a gate that sees only SAM3-side features and SAM3's confidence reaches +2.72 [2.25, 3.20] while switching only 11.7% of the time, matching the dense gate (+2.44). Without SAM3's confidence the same gate gets +0.81. So most of the value is knowing when SAM3 is unsure and using a second detector-grounded mask then; we cannot claim it is relational reasoning. We will say so in the paper.

We thank the authors for their response and corrections, which improve the clarity of the manuscript. However, the overall contribution of this work remains limited. For instance, the SAM3 already suffices in most scenarios, and the proposed gating mechanism yields only marginal improvements over it. Given these limitations in both architecture and performance, I maintain my initial score.

Reviewer_7cgvOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ReSCU constructs a scene graph and interprets a referring expression as a set of constraints to produce a symbolic segmentation candidate. An arbitration mechanism selects between this candidate and a SAM3 text-conditioned prediction. The paper compares hand-set and learned arbitration gates, reports improvements and transfer for the learned gates, and analyzes where the two branches succeed or fail.

Critical strengths

[s1] Using explicit relational information to complement an appearance-based prediction is well motivated. The reported improvements from the learned gates and the transfer results suggest that the symbolic branch can provide useful information beyond the baseline.

[s2] The complementary-error and failure analyses help explain when arbitration succeeds and where the pipeline remains limited. These diagnostics are useful beyond the final performance comparison and provide a substantive reason to pursue the approach.

Critical weaknesses

[w1] The paper does not consistently specify the method behind the main comparison. The method description refers to full-expression input to SAM3, while the baseline description refers to a head noun phrase. This distinction directly affects a claim about relational understanding. The soft-constraint equation also appears to remove a relation's contribution when the softness flag is enabled, and the supervision used to fit the learned gates needs clarification. Until these points are resolved, it is difficult to determine exactly what the reported comparison establishes.

[w2] The selection history limits the interpretation of the evaluation as independent confirmation. The appendix states that earlier A/C configurations were selected with access to test performance. Repeating selection using validation data is a useful check, but does not undo the earlier exposure to test results. The authors' disclosure is welcome; the remaining issue is which results support generalization without having influenced development or configuration selection.

Questions for authors
  1. Clarification: Do the SAM3 baseline and the appearance branch receive exactly the same full expression? If not, is there a comparison with matched inputs, and does the advantage remain?

  2. Clarification: Which reported results come from data that had not been inspected during method development or configuration selection? Please distinguish these from results obtained by rerunning selection after test or transfer performance had already been observed.

  3. Clarification: What is the implemented soft-constraint score, and what training targets and annotations are used to fit the learned gates? Please reconcile these with the equation and the stated zero-shot setting.

2 comments

Thank you for the careful and fair review; you correctly identified issues we had not resolved. We agree with s2 (the complementary-error and failure analyses are the useful part), and the new audit extends them.

w1 (specification). (a) SAM3 input: both the baseline and the appearance branch receive the full expression in all 7,596 val+test rows; the head-noun wording in Sec. 4.1 / App. F is wrong. (b) Soft-constraint equation: you are right. Eq. 3 corresponds to the first scoring stage; with soft ranking on, a second re-ranking stage re-sorts every candidate, and the first stage's relation terms (edge bonus, margin, −0.22-0.22 penalty) have no effect on the output. On five worked relation examples, ignoring the first-stage order changed the final top-1 in 0/5; the re-ranker's geometry term changed it in 3/5. We will replace Eq. 3 with the implemented re-ranker score, s(x)={[0.52cos⁡(x,e)+0.48cos⁡(x,t)] hlabhlay+0.20 mˉ(x)}⋅γs(x)=\{[0.52\cos(x,e)+0.48\cos(x,t)]\,h_{lab}h_{lay}+0.20\,\bar m(x)\}\cdot\gamma, with γ\gamma = 0.55 if a geometric predicate fails. Because the change is in the description, no reported number changes. (c) Gate supervision: label y=1y=1 iff IoU(symbolic mask, GT) >> IoU(SAM3-text mask, GT), from 2,573 RefCOCOg-val first-sentence objects, so ground-truth masks are used. The system is therefore not zero-shot, and we will say so.

w2 (selection history). Thank you for the clear framing. Our accounting:

ResultSelected without test/transfer access?
Sparse (logistic) and dense (boosted) gates, GDINO-rebuiltYes (selection logs contain no test/transfer fields)
Same protocol on Faster R-CNN dumps (the submitted tables)The protocol had earlier been run with test results seen; not test-naive
Threshold gate (lock A) and lock BNo: test deltas entered the choice
Earlier dense variant that used RefCOCO/+ val in selectionNo: appendix only
Coverage curve vs. thresholdPost-hoc on test, descriptive
Internal constants (priors, softmax temperature, 1.516 promote score, re-ranker weights)Selection history not logged; we cannot certify them as test-free

We will add this table. We agree the reported gains are not fully independent confirmation. The cleanest evidence is (i) frozen transfer of the RefCOCOg-selected gates to RefCOCO/RefCOCO+ (all CIs exclude 0) and (ii) the new train-split gates (dense +2.95 [2.35, 3.56]; sparse +1.57 [1.09, 2.05]), whose only selected parameter is a threshold chosen on val.

Q1. Yes, identical input (full expression). Q2. See table above. Q3. See w1(b) and w1(c).

Thank you for the honest explanation and clarifications. However, I believe the paper still needs major revision before acceptance, and I maintain my original rating.

Reviewer_a6sTOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper addresses zero-shot referring image segmentation. The authors argue that the standard zero-shot approach, which ranks candidate masks by their CLIP similarity to the expression, breaks down on relational expressions such as spatial relations (“left of”, “between”) and negation. A pooled text embedding does not record where the target sits relative to the anchor. A masked crop of a single candidate cannot show it either. ReSCU makes these relations explicit. It build a scene graph over the detected proposals, and the expression is parsed into a typed logical query that is executed against the graph to select the mask. For each image, a calibrated gate picks either this symbolic mask or a default “appearance” mask, obtained by prompting SAM3 with the text.

Critical strengths
  1. The oracle analysis shows real headroom. The symbolic branch and SAM3 are strongly complementary. Always picking the better of the two would score 11.24 point above SAM3-text on RefCOCOg val, and the symbolic mask is the better one on about 36% of images in an 800-image slice. This matters because it’s the strongest evidence that explicit relational reasoning adds something SAM3 doesn’t have, even though the current gate captures little of it.
Critical weaknesses
  1. The experiments don’t test the paper’s own motivation, and the gains over SAM3 are small. The paper argues against CLIP retrieval, but it only strong matched baseline is SAM3-text. Lock A is at parity with it. The learned gates add +1.89 +3.38 test mIoU at about 14.6× the runtime. Stronger training-free baselines such as SAM 3 Agent are missing. The claim to lead all zero-shot methods mostly comes from SAM3 itself. This is the main reason for my score: the central claim, that relational reasoning closes the relational gap, isn’t shown.
  2. The zero-shot claim doesn’t hold as stated. Locks C and D are trained on RefCOCOg val and selected using ground-truth masks, and lock A thresholds were tuned on the same split. That contradicts “no component is fine-tuned” and makes the Table 2 comparison with annotation-free methods unfair. The paper also doesn’t say what the detector was trained on. If it was COCO, the detector has already seen box annotations for the RefCOCO images. Since the whole paper is framed as zero-shot RIS, this affects how every headline number should be read.
  3. the paper is sooooo hard to read because much of it reads like a revision log, with repeated references to “our original submission” and “this revision”. This history should be removed or moved to an appendix so the main text presents one clear system.
Questions for authors
  1. What exactly is SAM3-text given as input? section 4.1 says only the head noun phrase, while Sec. 3.6 says the full expression. Which is it? If it’s the head noun phrase, how does sam3 do with the full expression?
  2. What was the detector trained on? if it was COCO, how much does its training data overlap with the RefCOCO/+/g images?
  3. Where do the gains of locks C and D come from? How much of the +1.89 +3.38 test mIoU comes from Type R expressions? If most of it comes from non-relational expressions, the gains don’t support the relational-reasoning ciaim.
1 comment

Thank you. Your three weaknesses are fair, and we respond with new evidence. We also agree with your strength: the oracle analysis (SAM3 and the symbolic branch are strongly complementary) is the part of the paper we think survives; it is where we are re-centering the work.

W1 (baselines, small gains, central claim not shown). We ran the control the paper lacked: does relational reasoning produce the gain? On RefCOCOg test (GDINO dumps, full-expression SAM3 baseline 53.27, image-grouped paired bootstrap 95% CI):

VariantΔ\Delta vs SAM3-text
Dense (boosted) gate, full system+2.44 [1.79, 3.11]
Relations removed from the symbolic branch, gate frozen+2.42 [1.77, 3.09]
Relations removed, gate refit+2.72 [2.08, 3.39]
Gate on SAM3-side/text features only, no SAM3 confidence+0.81 [0.52, 1.10]
Same, plus SAM3's confidence (switches only 11.7%)+2.72 [2.25, 3.20]
Random promotion at the same 32.5% rate-4.16 ±\pm 0.30

You are right that the central claim "relational reasoning closes the relational gap" is not supported by our data. What the data do support: a second, detector-grounded mask is better than SAM3's on 37.9% of test images (oracle +11.43 [10.71, 12.17]); a gate recovers part of that (+2.44) while blind promotion hurts (-4.16). Our last control shows the gate does not even need the symbolic mask's features: SAM3-side features plus SAM3's own confidence give +2.72. So the gain is "fall back to a detector-grounded mask when SAM3 is unsure", and relational machinery is not shown to be necessary. We will rewrite the paper around this. Relations help only in a small subset: on directional relations (n=144, 2.9% of test) the symbolic mask beats SAM3-text (47.6 vs 43.1) and the gate gains +4.55 [0.7, 8.8]; this is exploratory. We did not run SAM 3 Agent; we will remove the claim to lead all zero-shot methods, and state that much of the gap to CLIP-based methods comes from SAM3 itself. The runtime overhead is real (about 14.6x in our timing on the Faster R-CNN configuration, in the paper's Limitations); both branches must be computed before any gate decides, so a cheaper gate does not remove it.

W2 (zero-shot). Agreed (see Comment 0). The gate's labels are y=1y=1 if IoU(symbolic, GT) >> IoU(SAM3, GT), computed on 2,573 RefCOCOg-val objects, so ground-truth masks are used. Detector data: the Faster R-CNN fallback (torchvision COCO_V1) was trained on COCO train2017, which contains all RefCOCO/+/g images (category boxes only, no referring expressions). GroundingDINO-T was trained on Objects365, GoldG and Cap4M. We will state this.

W3 (revision-log language). Agreed; we will remove it and present one system.

Q1. Full expression in every run; the head-noun wording was a text error (Comment 0). Head-noun baseline (H100) is weaker or equal: val 50.57 vs 53.60, test 52.43 vs 53.27.

Q2. See W2. Overlap with COCO is complete for the fallback detector.

Q3 (where the gain comes from). Per expression type on RefCOCOg test (paper, FRCNN dumps, dense gate vs SAM3): S +3.7 (n=3,220), R +2.8 (n=1,388), N +1.2 (n=78), C +2.6 (n=337). So most of the gain is on non-relational types, which is consistent with your concern. In the audit round, directional relations (left/right/above/below; n=144) show a larger gate gain (+4.55 [0.7, 8.8]) than expressions with no parsed relation (+2.24 [1.6, 3.0]), but they are only 2.9% of test expressions and the subgroup is small, so we treat it as exploratory.

Comments on the paper

1 comment
AuthorsOpenReview ↗

summary of responses and commitments

e thank all four reviewers. Several of you independently found the same problems. We audited our code, ran new controls, and found that every concern about specification, fairness and selection was correct. We answer them directly rather than defend the original framing. We cannot upload a new PDF at this stage, so the changes below are commitments for the revised version; the new numbers are given here so you can check them now.

Verified by code audit and new experiments

  1. SAM3 input (a6sT, 7cgv, AAAb, jdbt). In every reported run the appearance branch and the "matched SAM3-text" baseline receive the full referring expression (7,596/7,596 dumped rows). The "head noun phrase" wording in Sec. 4.1 / App. F is an error in the text, not a different baseline. As an extra baseline (H100), a spaCy head-noun-phrase prompt is weaker or equal (full minus head: val +3.03 [1.41, 4.69]; test +0.84 [-0.25, 1.96]), so the full expression is the stronger comparison. The gates still beat the head baseline (test: sparse +2.44 [1.31, 3.54], dense +3.28 [2.14, 4.42]).
  2. Equation vs. code (7cgv). Right: the soft-constraint score in Eq. 3 does not determine the symbolic output under our setting; a second re-ranking stage does. We will describe the score that is actually implemented.
  3. Not zero-shot (a6sT, jdbt, 7cgv). Agreed. Lock C/D are fit on RefCOCOg-val using ground-truth masks to build labels. We will drop "zero-shot" and describe the system as a training-free pipeline with a small gate fit on a held-out split.
  4. Where the gain comes from (jdbt, AAAb, a6sT). New controls show the gain does not require explicit relations, and that a gate seeing only SAM3-side features plus SAM3's own confidence matches the full gate (+2.72 [2.25, 3.20] vs +2.44 [1.79, 3.11]). Our honest reading: the gain is falling back to a detector-grounded mask when SAM3 is unsure, not relational reasoning. We will rescope title, abstract and claims accordingly.
  5. Gate trained on a train split (jdbt). Done: fit on 5,000 RefCOCOg-train expressions, threshold chosen on val, test once: dense +2.95 [2.35, 3.56], sparse +1.57 [1.09, 2.05]; frozen transfer to RefCOCO/RefCOCO+ is positive for both.

Commitments for the revised version: one definition of SAM3-text; the implemented score instead of Eq. 3; a table marking which results were selected without test access; the controls above in the main text; removal of "zero-shot" and "leads all zero-shot methods" claims; removal of revision-history language; the three gates (threshold, sparse, dense) presented once in one section; two or three contributions; a vector pipeline figure; a statement of detector training data. We will also run ReferIt and a stronger training-free baseline (SAM 3 Agent) and report them whether or not they help.

What we ask. Every point of confusion or unfairness you raised (SAM3 input, equation vs. code, zero-shot, selection history, missing ablations) turned out to be correct, and we have fixed or measured each one rather than argued it. The corrected paper makes a narrower claim: a learned gate gives a small but significant gain over SAM3 that replicates under train-split fitting and transfers frozen to RefCOCO/RefCOCO+; the gain comes from a confidence-triggered fallback to a detector-grounded mask; explicit relations help only on a small directional subset. We would ask you to judge the paper on that corrected, fully disclosed claim, and we welcome any further control you would like to see before the discussion closes.

We reply under each review with specifics.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.