acceptodds
Under review as a conference paper at ICLR 2027

InstanceBench: Diagnosing Referential Reasoning and Target Identity in Referring Expression Segmentation

Abstract

Referring Expression Segmentation (RES) links natural-language descriptions to pixel-level object masks. Yet standard evaluation provides limited insight into instance-level referential reasoning: it does not systematically distinguish referential logics, test target preservation across valid grounding paths, or separate target-selection from mask-generation errors. We introduce InstanceBench, an instance-centered diagnostic benchmark comprising 6,194 images, 9,264 target instances, and 25,077 human-verified expressions. Each target-centric expression set (TCES) fixes the image and target mask while pairing a minimal expression with a same-target variant that uses another valid cue or grounding path. A compact referential-logic taxonomy spans direct target evidence, same-class selection, relational and compositional grounding, and exclusion, while logic-critical construction suppresses simpler shortcuts. Identity-aware metrics measure target retention and set-level success while separating selection from mask-generation errors. Across 22 native-mask RES checkpoints from 18 model families, the strongest checkpoint reaches 67.1% mIoU but only 59.6% All\@0.7. Controlled interventions confirm language sensitivity, while failure decomposition identifies target selection rather than mask decoding as the main bottleneck. On a controlled training subset, matched supervision improves identity-aware performance, showing that the diagnosed capability responds to targeted supervision. Collectively, InstanceBench supports a measure–diagnose–improve cycle: measuring target consistency across grounding paths, localizing failure sources, and evaluating targeted interventions.

Primary area:
datasets and benchmarks
Keywords:
Referring Expression Segmentation, Multimodal Reasoning, Dataset and Benchmark

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_rZy6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces InstanceBench, an instance-centered diagnostic benchmark for referring expression segmentation (RES). It is built around target-centric expression sets (TCESs), each of which pairs multiple expressions referring to the same target instance through different referential logics. The authors evaluate a diverse set of RES model families and MLLMs and report several diagnostic findings, revealing gaps in reliability and target-identity preservation.

Critical strengths
  • The proposed target-centric expression sets provide a structured way to evaluate whether models understand different referential logics.

  • The paper presents a comprehensive evaluation of existing RES models and MLLMs and delivers interesting findings.

Critical weaknesses
  • Closely related benchmarks such as OmniLabel and D³ (Described Object Detection) are not discussed. Both evaluate grounding of free-form descriptions, including negation and relational expressions. The authors should discuss these benchmarks and clarify InstanceBench’s contributions relative to them.

  • The paper proposes new metrics such as All@τ, All-target-ID, and NCI, but it is not clear why they are important or what additional information they provide beyond standard metrics. For example, All@τ is by construction upper-bounded by per-expression accuracy. The authors should demonstrate that these metrics reveal insights that conventional metrics such as mIoU or P@τ cannot.

  • The expressions derive from GPT-5.5 proposals, which may introduce linguistic or distributional bias. The authors should analyze this, for example by comparing performance on human-written versus model-generated expressions.

  • The paper reports several diagnostic findings, but it does not establish that they are new and could not have been observed with existing benchmarks.

Questions for authors
  • How do the authors ensure that there is no train/test contamination, given that the COCO-derived images and masks come from existing benchmarks?
  • How do the authors ensure that the reported findings are new and not observable in existing benchmarks?
1 comment

Dear Reviewer rZy6,

As the discussion period is drawing to a close, we wanted to check whether you have any remaining concerns that we could further address.

If our responses have adequately addressed your concerns, we would greatly appreciate your consideration of a score update.

Thank you again for your thoughtful feedback and engagement!

Reviewer_2yiaOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces InstanceBench, an instance-centered benchmark for RES that systematically evaluates referential reasoning, target consistency across valid grounding paths, and target-selection vs. mask-generation errors. Through extensive evaluations of RES checkpoints and controlled supervision experiments, the authors identify target selection as a major bottleneck and demonstrate the benefits of targeted supervision.

Critical strengths
  1. The task is clearly defined, and the evaluation protocol aligns well with the objectives.

  2. The experiments are extensive and provide substantial support for the benchmark’s conclusion.

Critical weaknesses
  1. Conditional mask IoU measures mask stability rather than mask accuracy. The paper reports conditional mask IoU between minimal expression and its same-target variant. However, both predictions may omit the same object region. High agreement between predicted masks does not mean accurate mask prediction compared with GT. An experiment that reports GT IoU/boundary metrics conditioned on correct target selection can better support the conclusion that target selection is the bottleneck.

  2. The matched-supervision study does not fully isolate the logic-aware supervision from linguistic diversity/distribution adaptation. The source-text control replaces track expression with its RefCOCO-family expression, which may repeat the same expression across multiple rows, while logic-aware supervision retains distinct human-verified track expressions. Therefore, the improvements may come from linguistic diversity, and closer alignment with the test distribution. Matching the number of unique expressions per TCES, and comparing supervision using same-logic paraphrases and alternative-path variants would be more compelling.

Questions for authors

Please refer to the weaknesses for details.

1 comment

Dear Reviewer 2yia,

As the discussion period is drawing to a close, we would like to sincerely thank you for your positive assessment, constructive suggestions, and support of our work.

We greatly appreciate your thoughtful feedback, which has helped us further improve the clarity and quality of our paper.

Thank you again for your time and consideration!

Reviewer_6eUaOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces InstanceBench, a referring-segmentation benchmark of 6k COCO/OpenImages images and 25k expressions. Each target-centric expression set (TCES) fixes one target mask and pairs a minimal expression with a variant that identifies the same target through a different cue. GPT proposes expressions under a nine-category logic taxonomy, and human annotators revise, replace or add to them. The paper adds set-level metrics ([email protected], All-target-ID), a normalized consistency index (NCI), and a split of failures into identity and mask errors. Validation covers 22 checkpoints, text interventions, and a small supervision study. It concludes that average scores hide unreliable targets, that target selection rather than mask decoding is the main bottleneck, and that the capability responds to logic-aware training.

Critical strengths
  1. The design. Fixing the image and mask while changing the cue allows a comparison that existing RES benchmarks don't really offer, and it takes real annotation effort. For the four models tested, category-only prompts fall, so language is needed on this data.

  2. Care in construction and reporting. The prompts, a blinded audit and an anonymous release are all provided. Counts reconcile across the paper: I rebuilt Tab 9 from Fig. 7 and Fig. 4(b) from Tab 16, and both match. The authors also say plainly that Tab. 2 is a transfer comparison and that this is not a natural distribution. This is why I take the dataset seriously even though I disagree with some conclusions.

Critical weaknesses
  1. The supervision study doesn't show that the capability "responds to targeted supervision". The source-text control is probably, in large part, text these RefCOCO-trained checkpoints were already fitted on, so the contrast may be old text vs. new text rather than logic-aware vs. not. There is no arm with new text that isn't logic-aware, and the treatment comes from the same authoring pipeline as the test expressions, so the gain may just be alignment to the benchmark's style. No per-logic gains are shown, which "targeted" needs. Only LAVT and DETRIS are tested, for three epochs. This matters because it is the "improve" half of the paper's main story.

  2. "Target selection rather than mask decoding" is broader than the analysis behind it. Conditional mask IoU is IoU between two predictions, so it measures agreement, not quality: PaDT has 94.56 in Table 3 but only 3.36 [email protected] in Table 2. In Fig. 4(b), identity errors are 73-81% of failures but confirmed switches are only 53-58%, so more than a quarter of that bucket isn't a confirmed wrong-instance selection. The analysis also covers only five universal models at IoU 0.7, while the abstract states it generally. I expect the direction to hold for these models, but not the general claim.

Three smaller points push the same way. Sec. 4.3 itself says the multi-anchor effects aren't statistically resolved after matching, yet relational composition is still listed as weak. The drop in retention and the lower NCI for variants are also largely expected, since a paraphrase is nearly the same input and RES models treat each expression independently. Finally, every expression refers to exactly one present target, while several recent benchmarks cover multi-target or no-target cases (gRefCOCO, MMR, HalluSegBench). A model is never penalized for always returning a mask, so "reliability" here doesn't include rejecting a false description.

Questions for authors
  1. Did you run a supervision arm with new text that isn't logic-aware? What are the per-logic gains, and which test set does Table 8 use? Its levels look like the full test set, while the caption says COCO-derived subset.
  2. What is IoU to GT given correct ID, by model group and at both 0.7 and 0.9?
  3. What is inside "unresolved": merged masks, empty masks, other-class masks, or instances missing from the inventory? How does it split by source?
  4. Which RefCOCO split defines the test set, and how many test images are in each checkpoint's training data?
  5. Why is oracle candidate-ID below direct box prediction for all five models in Tables 6 and 7, if it isolates selection?
  6. Logic-wise All-target-ID per source, and retention per minimal-to-variant transition using Fig. 9(b).
  7. Scores on the original RefCOCO expressions for the same COCO targets, to separate logic difficulty from text style.
  8. Could TCESs and All-target-ID be extended to no-target and multi-target cases, for example by scoring rejection of false descriptions and matching predicted sets to targets?
1 comment

Dear Reviewer 6eUa,

As the discussion period is drawing to a close, we would like to kindly follow up on our rebuttal and invite you to take a look at our responses to your comments.

We greatly appreciate your feedback and are eager to address any remaining concerns or questions you may have. We would be grateful if you could let us know whether any points require further clarification.

Thank you again for your time and consideration!

Reviewer_ic5DOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

paper claim that standard RES evaluation scores each expression–mask pair independently, and that mask overlap conflates selecting the wrong instance with producing a poor mask. It introduces InstanceBench: in scenes with at least four same-class instances, a fixed target is described by a minimal expression and by a variant that uses a different referential logic. the authors use a 9 category logic taxonomy and identity-aware metrics All-target-ID, retention, NCI. They test which logic model execute reliably and whether target identity is preserved across grounding paths. They also test whether failures come from target selection or mask generation.

Critical strengths
  1. InstanceBench is a useful resource. The release includes the fixed-target expression sets with logic labels, the same-class candidate inventory and the evaluator. The constructions and audit are documented in detail. The selection mask decomposition is a simple tool that other res work could adopt directly.
  2. per-logic profiles are informative. The ordinal–extremal gap survives matched controls. Both the ordinal–extremal and nested–single-anchor gaps hold for every model on COCO and OpenImages. the oracle candidate-ID result shows clear room for improvement even for strong mllm.
Critical weaknesses
  1. The headline set-level gap holds by construction. Each TCES has two core expressions, each predicted from a single image–expression pair, so [email protected] can never exceed [email protected]. The gap between SAMTok’s 67.1 mIoU and 59.6 [email protected] would therefore appear for any imperfect models. And 59.6% is actually above the independence level of ([email protected])² = 50.7%.
  2. Consistency across grounding paths is the most distinctive claim but the one analysis of it lacks an informative reference. The paraphrase p keeps both the logic and the grounding path of m, so a high NCI(m, p) (0.565 to 0.770) work more like a ceiling than a baseline. The variant v uses different visual evidence by design, so a negative ΔNCI shows only that errors are cue-specific, not that models lack something an ideal referrer would have.
Questions for authors
  1. How were the paraphrases produced and verified, and how much do they actually differ from m? appendix D gives prompts only for the minimal expression and the variant, yet this track is the reference for the main consistency claim.
  2. How do the set-level results compare with what per-expression or per-logic accuracy alone would predict, instead of with mIoU? For NCI, is the m–v coupling clearly below a reference that changes the cue but stays in the same logic family?
1 comment

Dear Reviewer ic5D,

As the discussion period is drawing to a close, we would like to sincerely thank you for your positive assessment, constructive suggestions, and support of our work.

We greatly appreciate your thoughtful feedback, which has helped us further improve the clarity and quality of our paper.

Thank you again for your time and consideration!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: thank you to all reviewers

We thank all four reviewers for their careful and constructive feedback. We appreciate their recognition of InstanceBench’s target-centric evaluation design, referential-logic taxonomy, broad model coverage, and identity-aware diagnosis. The discussion helped us clarify these contributions and strengthen their empirical support.

InstanceBench is not simply another benchmark containing relational or negative descriptions. Its novelty lies in organizing referential reasoning into explicit logic families and evaluating segmentation and target-identity preservation through controlled fixed-target sets in dense same-class scenes: each TCES pairs expressions that reach the same target through different valid grounding paths. This design enables per-logic diagnosis, exact whole-TCES reliability measurement, matched analysis of how grounding-path changes affect target identity, transition-specific retention, and identity-versus-mask failure analysis. These capabilities are not isolated by the standard protocols of conventional RES benchmarks. These controlled same-target comparisons, rather than the general observation that relational expressions are difficult, constitute the central novelty of InstanceBench (rZy6 W1/W4/Q2).

The discussion led to the following main clarifications and supporting analyses.

  1. Metric interpretation and consistency analysis. We clarify that [email protected] measures exact whole-TCES reliability rather than statistical independence, and report role-specific independence references and matched NCI controls for wording changes, cue changes, and cross-family path changes (ic5D W1/Q2(a), W2/Q2(b); rZy6 W2).

  2. Expression construction and language controls. We clarify paraphrase construction and verification, compare retained GPT-proposed/human-verified expressions with human-written expressions, and evaluate original RefCOCO expressions on matched targets to examine possible effects of language source and text style (ic5D Q1; rZy6 W3; 6eUa Q7).

  3. Failure analysis. We report GT mask quality conditioned on correct target identity and further separate confirmed same-class switches, unresolved assignments, and correct-ID predictions with insufficient mask IoU. Among single-anchor and compositional predictions with GT IoU below 0.7, confirmed same-class switches remain the largest individual failure category across all model groups. A complementary decomposition across all five high-level logic families in Appendix H shows the same pattern in every family, and the high-threshold analysis clarifies the remaining role of precise mask delineation (6eUa Q2/W2, Q3; 2yia W1).

  4. Supervision study. A matched new-text control compares same-path paraphrase supervision with alternative-path supervision under identical data volume and training settings. Alternative-path supervision improves [email protected] and All-target-ID across all five logic families (6eUa Q1/W1; 2yia W2).

  5. Data provenance and evaluation scope. We clarify source provenance and upstream checkpoint exposure, report identity results by source and grounding-path transition, clarify the candidate-inventory ID diagnostic, and discuss extensions to no-target and multi-target settings. Removing the union of all 185 audited overlapping images leaves the main comparative conclusions unchanged (6eUa Q4–Q6, Q8; rZy6 Q1).

We have uploaded a revised manuscript, with revised material highlighted in blue. Detailed results and answers are provided in the response to each reviewer, respectively; all section, table, figure, and appendix references refer to the revised manuscript. We believe these clarifications and additional analyses address the reviewers’ main concerns and strengthen the evidence for our central claims. We welcome further questions and would be happy to provide additional evidence or clarification where useful.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.