acceptodds
Desk rejected at ICLR 2027

MedCounterSeg: Benchmarking Unsupported Medical Queries and Learning to Defer through Concept-Based Visual Reasoning

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Recognizing a medical concept does not establish that a particular image supports it. A segmentation model must distinguish these judgments to avoid turning an unsupported query into a plausible mask. We introduce the Medical Query-Counterfactual Segmentation Benchmark (MedCounterSeg-Bench): 6,018 supported–unsupported query pairs across six imaging modalities, 42 concepts, and five clinically reviewed mismatch levels comprising ten subtypes. Each pair shares a target image, while each query receives valid positive examples of its requested concept. The benchmark separately measures concept localization, target-support decisions, and mask quality, with clinician involvement and iterative human quality control during construction. We further propose MedCounterSeg-RL, which learns to defer through concept-based visual reasoning. After positive-only concept–mask alignment, reinforcement learning jointly trains segmentation and deferral; correct deferral retains credit for locating the concept in a positive verification image. With Qwen3.5-4B, the method achieves the highest paired accuracy among evaluated methods on both the in-distribution Test split and unseen datasets, with segmentation quality close to ConceptSeg-R1 on in-distribution queries. The benchmark isolates fine-grained lesion and anatomical mismatches as the main remaining challenge.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Medical Image Segmentation, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_G3pFOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MedCounterSeg-Bench, a benchmark comprising 6,018 supported–unsupported query pairs across six medical imaging modalities and five mismatch levels. The benchmark distinguishes concept recognition, target-image support decisions, and segmentation quality through paired queries and positive concept references. The authors further propose MedCounterSeg-RL, which learns segmentation and deferral jointly through reference-based visual reasoning and verification rewards. Experiments show improved overall paired accuracy over existing rejection-enabled methods, while revealing substantial challenges in fine-grained medical concept discrimination.

Critical strengths

Clear problem decomposition and meaningful benchmark design. The distinction between recognizing a medical concept and determining its presence in a particular image is well motivated. The paired-query protocol, positive concept references, and graded mismatch taxonomy provide a more informative evaluation framework than conventional rejection accuracy.

Dedicated method design aligned with the problem. MedCounterSeg-RL explicitly connects concept verification with segmentation and deferral. The verification and negative-mask rewards are well motivated, and the ablations provide useful evidence for their respective contributions.

Critical weaknesses

Severe benchmark imbalance and insufficient fine-grained evaluation. Levels 1–2 account for 75% of the benchmark, while Levels 4–5 contain only 181 and 201 pairs across all splits, with approximately 17 and 19 pairs in the in-distribution test set. The proposed method achieves 96.1%/95.2% paired accuracy on Levels 1–2 but only 23.5%/5.3% on Levels 4–5, underperforming SESAME and MedVeriSeg on the latter. Consequently, the overall improvement is dominated by coarse mismatches, while the most challenging distinctions remain insufficiently evaluated.

Performance claims and methodological contributions need stronger validation. The reported segmentation quality is close to ConceptSeg-R1 on accepted positive queries, but accounting for false deferrals reveals Dice reductions of 5.66 points on Test and 15.09 points on OOD. OOD paired accuracy also varies substantially across backbones (52.50% vs. 7.50%), while verification IoU remains relatively low (25.25%). Moreover, the proposed method uses additional reference images unavailable to rejection-enabled baselines, and the current ablation removes references and structured reasoning simultaneously. These factors make it difficult to isolate the contribution of the proposed reasoning mechanism.

Questions for authors
  1. Benchmark imbalance and fine-grained evaluation. What motivates the severe imbalance across mismatch levels? Given the limited L4–L5 test samples and the underperformance against existing baselines, how well does the benchmark support conclusions about fine-grained concept discrimination? Macro-averaged results and confidence intervals would provide a more balanced assessment.

  2. Performance and generalization. How should the substantial segmentation performance degradation after accounting for false deferrals be interpreted, particularly on OOD data? What explains the large OOD performance gap between backbones? Furthermore, does the relatively low verification IoU (25.25%) provide sufficient evidence of concept recognition during deferral?

  3. Reference images and reasoning. A reference-augmented comparison with SAM3-Agent/MedSAM3-Agent would help assess the contribution of additional visual information. More importantly, what is the independent contribution of explicit structured reasoning? An ablation retaining reference images and verification supervision while removing the reasoning fields would help clarify this.

  4. Training details. Please clarify the baseline training protocols, particularly the lack of target-domain adaptation for SESAME, and the composition of the 7,709 Stage I positive samples.

Reviewer_nPfROpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines whether a medical segmentation MLLM can distinguish between knowing a concept and finding evidence for it in a given image. It introduces MedCounterSeg-Bench, a benchmark of 6,018 supported/unsupported query pairs built from a subset of IMed-361M (62 source datasets, six modalities, 42 concepts). Both queries in a pair share the same target image, and each query comes with three boxed reference images and one unboxed verification image of the requested concept drawn from other cases. Unsupported queries are organized into five mismatch levels, ranging from modality and anatomical-site mismatches to slice/organ, lesion-context, and sequence/substructure/laterality mismatches. The evaluation separately reports mask quality, SEGMENT/DEFER decisions through action accuracy, paired accuracy, and correct and false deferral rates, verification-box IoU, and raw foreground probability before gating. The paper also proposes MedCounterSeg-RL, which couples a Qwen3.5-4B MLLM with a Concept Prompt Adapter that feeds frozen SAM 3. Training has two stages, a positive-only adapter alignment followed by GRPO with action, verification, mask, and format rewards, where the verification reward keeps concept-localization credit alive when the model correctly defers. On the in-distribution Test split, the model reaches 83.17% paired accuracy versus 65.17% for SESAME, and 52.50% on a CT-only OOD split built from AMOS2022 and WORD versus 30.00% for MedVeriSeg. Reward ablations and target-case bootstrap intervals are provided.

Critical strengths
  1. The paired, same-image evaluation protocol with a five-level mismatch taxonomy is carefully designed. Requiring a correct decision on both queries of the same image exposes the "refuse everything" shortcut that a single rejection rate hides (SESAME reaches 91.5% correct deferral but 29.3% false deferral, Table 2), and reporting verification-box IoU and raw foreground probability before gating makes the decision and the mask separately inspectable.
  2. The verification reward is a simple and sensible mechanism. A correct deferral still earns credit for localizing the concept in a valid example, so the policy is not pushed toward rejection by losing all localization signal.
Critical weaknesses
  1. The benchmark's label definition, level composition, and human review are not documented well enough to tell what the headline numbers measure.

    • The label definition and the corresponding benchmark construction do not align well. Section 3.1 defines the label in terms of visual evidence, with y = 1 meaning "sufficient visual evidence to localize the requested concept" and deferral meaning "insufficient evidence in the image, not clinical absence in the patient." However, by construction, y = 1 if and only if a mask exists in the source annotation, and y = 0 if and only if a rule places the concept outside the image (wrong modality, wrong site, absent on the slice, a different lesion or substructure). The two agree on y = 0, since absence implies no evidence, but not on y = 1, since an annotated object need not be evaluable on the sampled 2D slice, and no screening for evaluability is reported. A lesion annotated from the volume but barely visible on the slice, or a low-quality fundus or endoscopy frame, can be y = 1 by construction and y = 0 by definition, and a clinically correct deferral on it is scored as a false deferral. This affects lesion concepts most, which take a large share of the supported queries (polyps, edema, skin lesions, tumors).
    • Sample counts per level per split are never reported. Table 6 gives split totals and Figures 4-7 pool all splits, so Table 10 and the per-level claims in Section 5.2 are percentages over unknown denominators. If splits are drawn randomly without stratification, nearly 75% of the test pairs are trivial modality (L1) or anatomical site (L2) mismatches, such as asking for a skin lesion in a dermoscopic view on a brain MRI in Figure 12. These are not the plausible hypotheses the introduction motivates, and SESAME rejects over 98% of them with natural-image weights. From Table 10, the clinically interesting L4/L5 levels seem to have fewer than 20 pairs, so the claim that no compared method exceeds 58.8% and 42.1% could be statistically unstable. The OOD split (160 pairs from 35 CT cases, organ concepts seen in training) is similarly too small to support the "unseen datasets" claim. Moreover, the Qwen2.5-VL-3B variant collapses drastically to 8.75% correct deferral on it (Table 7).
    • Human review is qualitative rather than quantitative. Appendix A.5 describes an iterative collaboration of meetings, 100-sample batches, and rule revisions, but gives no review protocol, no judging criteria, no number of clinicians or rounds, no agreement, and no error rate on the reviewed pairs. The meaning of "reasonable and reliable" is not clear, and the "endorsed" qualitative examples cover only 19 author-selected cases. "Clinically reviewed" nonetheless appears in the abstract, Table 1, and the contribution list.
  2. The paper does not establish the necessity of an MLLM trained with RL for this task, and the experiments cannot fully attribute the gains to the proposed training.

    • As constructed, the decision task is few-shot concept-presence classification, where the model is given exemplars and a target, decides whether the concept is present, and provides prompts for SAM 3 to segment. No non-MLLM baseline is included, for example, a frozen encoder or SAM 3 exemplar features with an action head trained supervised or with the same action reward. The concept-based reasoning that motivates the MLLM has no training signal, since Section 4.3 uses no reasoning traces, and it has no quantitative evaluation anywhere in the paper. The only related ablation removes references and reasoning fields at inference from a model trained with them, which is a train/test mismatch rather than evidence that reasoning helps. The cost of this design is also unexamined. Each query runs a 4B MLLM over four-image references plus the target, and then runs SAM 3, yet no parameter counts, latency, or token budgets are reported for any method, so the accuracy/compute tradeoff against a lighter decision model cannot be judged.
    • The baselines are not matched. Only the proposed method and ConceptSeg-R1 receive the reference set. Among the decision-enabled baselines, only GSVA with LLaVA-7B is fine-tuned on the same data, and the rest use released weights or run training-free. The controls that would attribute the gain are missing, such as Qwen3.5-4B without Stage II training but with the same reference input, same-backbone SFT with a rejection token on the same queries, and ConceptSeg-R1 with a rejection threshold on foreground probability. All results come from one training run, and ablation deltas (e.g., 83.17 vs 81.00 paired accuracy) fall inside the full model's own bootstrap interval (79.15 to 86.97).
    • Table 2 reports Dice over each model's accepted queries, as Table 11 suggests. On all supported queries, the proposed model is at 64.69 Dice versus 70.35 for ConceptSeg-R1 with the same backbone (paired difference -5.66, 95% CI [-8.28, -3.07]), and on CT at 55.2 versus 73.7 (51 of its 56 false deferrals occur on CT). The accepted-only choice is understandable, since a false deferral yields an empty mask and a Dice of 0, but it should be stated early in Section 5.1 and Table 2 with both metrics reported, and the abstract's claim of segmentation quality close to ConceptSeg-R1 should be qualified.
  3. Terminology is inconsistent, which is less consequential but worth fixing. "Deferral," "rejection," and "support decision" are used interchangeably. The task definition says DEFER means insufficient evidence, Figure 2 labels the negative query as "requested concept absent," and Figure 8 calls paired accuracy "hallucination resistance."

Questions for authors

On the benchmark (Weakness 1):

  1. Could you report the number of pairs per mismatch level and per modality for the training, validation, Test, and OOD splits?
  2. How are labels verified? For y = 0 in L3/L4, is absence inferred from the absence of an annotation in the source dataset, and how are datasets that annotate only a subset of structures or lesions handled? For y = 1, was any screening applied to confirm that the sampled 2D slice actually shows the annotated concept, such as contrast phase, windowing, artifacts, or fundus and endoscopy quality, and is a "not evaluable" image labeled or excluded?
  3. Human review. How many clinicians, which specialties, how many rounds, how many examples in total, what exactly were they asked to judge, and did you measure agreement or an error rate?

On the method and comparison (Weakness 2): 4. Can you report the missing controls, namely a non-MLLM decision baseline with an action head on a frozen encoder or SAM 3 exemplar features, Qwen3.5-4B run zero-shot with the same prompt and reference set, and a same-backbone SFT model with a rejection token on the same Stage II data? Multiple Stage II seeds for the full model and the ablations would also help. 5. Why does Stage II use only 800+800 of the 4,697 training pairs, and does L3-L5 performance change with more data or updates?

Reviewer_PejaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces MedCounterSeg-Bench for evaluating whether medical segmentation models can distinguish supported from unsupported queries. It also proposes MedCounterSeg-RL, which combines reference-based concept reasoning with SAM 3 and reinforcement learning to jointly learn segmentation and deferral

Critical strengths
  1. The paper addresses an important problem: preventing medical segmentation models from producing misleading masks for unsupported queries.
  2. This paper curated a benchmark and includes extensive experiments covering multiple baselines, out-of-distribution evaluation, and component ablations.
Critical weaknesses
  1. The architecture largely follows the established MLLM–adapter–SAM paradigm, and the reference-based concept reasoning mechanism is not sufficiently differentiated from existing approaches such as ConceptSeg-R1. The main methodological contribution therefore appears to lie in the additional deferral-related objectives, particularly retaining concept-localization supervision after rejection. While potentially useful, this constitutes an incremental extension, and the paper does not adequately establish its advantage over a simpler support-classification objective within the same segmentation framework.

  2. Insufficiently controlled comparisons. Although the paper evaluates numerous baselines, two issues limit attribution of the reported gains. First, most decision-enabled baselines use older MLLM backbones, whereas the main model uses Qwen3.5-4B. It is therefore unclear how much improvement comes from the proposed method rather than the stronger backbone. Moreover, the same-backbone comparison in Table 2 shows no segmentation advantage over ConceptSeg-R1: the proposed method achieves slightly lower mDice (71.35 vs. 71.66) and mIoU (63.34 vs. 63.73) with Qwen3.5-4B.

  3. Second, most baselines do not receive the reference images available to the proposed method, and several are evaluated without task-specific training. These differences confound the comparison. A baseline using the same backbone, SAM 3, reference inputs, and training budget, but a simpler supervised deferral objective, is needed to establish the method’s independent contribution.

Questions for authors
  1. Can you replicate some of the baseline with same qwen3-4b backbone? I would be open to revising my current score if the authors provide stronger evidence through controlled comparisons that isolate the proposed method’s contribution from differences in backbone capability, reference inputs, and training settings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.