acceptodds
Withdrawn from ICLR 2027

Learning an Independent Multimodal Auditor for Language-Grounded Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Language-grounded segmentation models are increasingly used in multimodal systems, but they can produce confidently incorrect masks, including masks that select the wrong object entirely. This creates a distinct verification problem at deployment: given the image, referring expression, and a candidate mask, determine whether it covers the intended object accurately without seeing the ground-truth mask. A visually coherent segmentation can still be semantically wrong, so quality assessment must check the image–text–mask correspondence as well as mask geometry. Although much work improves mask generation, independent language-conditioned assessment of an already predicted mask remains underexplored. We introduce an independent multimodal large language model (MLLM)-based auditor that verifies predicted masks using only the image, referring expression, and mask itself, without access to ground truth or segmentation model-specific information. We formulate mask verification as estimating the quality of a predicted mask and show that language-conditioned attention provides a useful grounding signal to improve verification. The proposed auditor achieves a better balance between retaining correct masks and rejecting incorrect ones, outperforming existing mask-quality signals and learning-based baselines. It also generalizes across segmentation models and datasets, including challenging reasoning-based referring expressions. These results show that separating segmentation generation from verification provides an effective reliability layer for language-grounded vision systems.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Independent verification, Vision-language Models, Language-grounded segmentation, multimodal auditing, failure detection.

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_2TbQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The manuscript focuses on independent mask quality assessment for language-guided segmentation. It fine-tunes Qwen2.5-VL via LoRA to predict the ground-truth IoU from the original image, the referring expression, and a mask-overlay image, and introduces the proportion of mask-internal attention from selected attention heads as a prior. The model is trained on SAM3-I predictions over RefCOCO together with synthetic errors, and its quality estimation and accept/reject performance are validated on SAM3-I, X-SAM, and ReasonSeg, with additional text-substitution experiments examining language dependence.

Critical strengths

The research clearly distinguishes geometric mask quality from linguistic referring consistency. The cross-generator evaluation and the text-substitution experiments with fixed images and masks provide empirical support for the auditor's use of linguistic information.

Critical weaknesses
  1. Section 2 uses a 0.5 threshold on predicted IoU to decide acceptance or rejection, which has clear implications for an auditor trained under IoU supervision. However, Appendix A.2.2 directly uses the generator's native confidence, CLIP cosine similarity, and PMass without stating that they were IoU-calibrated. In Table 1, SAM3-I native confidence accepts all samples, while CLIP rejects all samples. This primarily exposes a mismatch between score scale and threshold; it cannot directly demonstrate that these signals lack validation ability. Table 4 also shows that SAM3-I native confidence achieves an AUROC of 0.809, indicating considerable ranking ability. A fair comparison should add in-training-set calibration, threshold selection on an independent validation set, and the false-acceptance rate at an equal rate of correct-mask retention. Table 4 does support that the auditor provides an improvement, but it cannot eliminate the risk that Tables 1–3 exaggerate the magnitude of the improvement; moreover, CLIP similarity does not appear in the AUROC table.

  2. The manuscript already notes that MQA-RefAVS has proposed candidate-mask quality assessment without ground-truth masks at inference time, IoU prediction, and training an MLLM auditor through supervised instruction fine-tuning. Therefore, "separating segmentation generation from quality auditing" and "the MLLM predicting the IoU corresponding to a ground-truth-invisible mask" are not in themselves novel contributions of this paper. The differences of this paper lie mainly in the image domain, cross-generator/cross-dataset evaluation, and the PMass prior; however, the related work only briefly states the difference in domain without adequately explaining the substantive methodological differences, and it provides neither an adapted direct comparison nor an explanation of why such a comparison cannot be made.

  3. Table 5 simultaneously changes the number of correct samples, the class ratio, the number of synthetic errors, and the overall training scale. Therefore, it cannot separately answer whether the benefit comes from class rebalancing or from specific synthetic errors. Moreover, the filtered training set yields decreased Macro F1 in both RefCOCO settings; the authors do acknowledge this trade-off, which is worth retaining — but it indicates that the strategy mainly shifts acceptance/rejection preference and cannot simply be regarded as an overall performance improvement. The supervised CLIP regressor uses the original training set while the auditor uses the filtered training set, which also adds confounding to the comparison. The attention prior lacks controls such as random scalars, mask area, or random attention heads, so the possible role of additional features or area correlations cannot be ruled out. The appendix's judgments that "aggregating content words is best" and "adding attention heads brings no benefit" are not accompanied by corresponding quantitative results.

  4. The cross-generator experiments cover only one unseen generator, X-SAM; the training source is only SAM3-I, and the auditor backbone is only Qwen2.5-VL. The evidence supports that "transfer occurs in these test settings," which is not yet sufficient to generalize to broad generator independence or robust deployment capability. In particular, Table 5 shows that the base auditor's Recall on the ReasonSeg test set is only 0.608, meaning it rejects 39.2% of correct masks. Combined with the bad-mask proportion of 46.47% and Specificity of 0.818 in Table 3, and assuming the same evaluation set and global statistics, roughly 20.6% of its accepted samples would still be bad masks. The 18.2% in Table 3 is instead "the proportion of bad masks that are accepted," and the two meanings differ. The paper does not provide a risk–coverage curve, the residual error rate of the accepted set, or actual downstream-task experiments; therefore, the deployment value of the "reliability layer" still lacks direct verification. IoU ≥ 0.5 is also merely the operational standard adopted in this paper, not a guarantee of the safety of downstream actions.

Questions for authors

Details can be found in the Critical Weaknesses section.

Reviewer_K7Z8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an independent auditor for language-grounded segmentation. Given an image, a referring expression, and a candidate mask, a LoRA-tuned Qwen2.5-VL-7B predicts the mask IoU without using the segmentation model's internal features. The paper also studies PMass, an attention-based score used as an additional prompt value. The auditor is trained on SAM3-I outputs from RefCOCO and evaluated on SAM3-I, X-SAM, and ReasonSeg.

Critical strengths
  1. The problem is practical. An auditor that is separate from the mask generator could provide a useful reliability check. The reported AUROC gains also transfer to X-SAM and ReasonSeg without further training.

  2. The text-swap experiment is a useful control. By changing only the referring expression while keeping the image and mask fixed, it provides direct evidence that the auditor uses the language input.

Critical weaknesses
  1. The main decision comparison applies the same 0.5 threshold to scores with different scales, including predicted IoU, native confidence, CLIP similarity, and PMass. This makes the reported bad-mask rejection and macro-F1 gains difficult to compare fairly. The AUROC results remain useful, but performance at matched recall or coverage is still unclear. At the reported operating point, the base auditor has 0.608 recall on ReasonSeg, so it rejects 39.2% of good masks.

  2. The source of the gains is not well isolated. The learning baselines use different backbones or training data, and PMass is not compared with a simple mask-area prior. On ReasonSeg, PMass improves decision metrics but worsens MAE from 0.263 to 0.270 and Pearson correlation from 0.515 to 0.496, so it is not a uniform improvement in continuous quality estimation. Generalization is also tested on only one unseen generator and one transfer dataset.

Questions for authors
  1. What are the results at matched good-mask recall or matched coverage, with all thresholds selected on a held-out training-domain split and then frozen?

  2. How does the CLIP regressor perform when trained on the same curated examples as the auditor? How does a same-backbone auditor perform without the referring expression?

  3. How does PMass compare with a simple mask-area scalar, shuffled PMass, or constant PMass? Can you also report uncertainty or multi-seed results for its gains?

  4. How does performance differ between geometric mask errors and semantic wrong-object errors? How many natural and synthetic training negatives belong to each group?

Reviewer_rugaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an independent auditor for language-grounded segmentation that estimates mask IoU using LoRA-tuned Qwen2.5-VL, with PMass incorporated as an attention-based grounding prior. The auditor takes an image, referring expression, and candidate mask without requiring ground truth at inference. Trained on curated SAM3-I masks from RefCOCO, it is evaluated in-domain and on X-SAM and ReasonSeg, reporting improvements in macro F1 and AUROC across the evaluated settings. In a text-swap experiment, replacing the expression with one referring to another object while keeping the image and mask fixed reduces the mean predicted IoU, and the resulting predictions correlate with the new referent’s true IoU.

Critical strengths
  1. A useful formulation of independent, language-conditioned mask auditing. Separating mask generation from quality assessment provides a useful way to examine both whether a mask selects the intended object and whether it adequately covers that object. The auditor requires only the image, referring expression, and candidate mask, allowing it to interface with existing segmentation systems without accessing their internal representations or native confidence scores. This gives the approach a clear practical motivation and a flexible input interface.
  2. Consistent empirical improvements across the evaluated settings. Trained only on SAM3-I masks from RefCOCO, the auditor demonstrates improvements on X-SAM outputs and ReasonSeg without adaptation. The attention-prior variant outperforms the reported baselines in macro F1 and AUROC across the evaluated in-domain, cross-generator, and cross-dataset settings, achieving AUROC values of 0.935, 0.827, and 0.775, respectively. The AUROC gains provide threshold-independent evidence of improved mask-quality discrimination within the tested settings.
Critical weaknesses
  1. Limited novelty. The cited MQA-RefAVS work already uses a multimodal model to predict mask IoU without access to ground truth at inference and addresses both semantic and geometric errors. Its method also employs LoRA-based instruction tuning. Therefore, this paper needs to explain more clearly the new challenges introduced by image-based auditing and the corresponding technical contributions, beyond outlining differences in input modalities and evaluation settings.
  2. The core pipeline substantially overlaps with MQA-RefAVS, and PMass is one of the most concrete additional components introduced in this paper. Experiments support the effectiveness of PMass as both a standalone quality signal and an additional auditor input. However, its advantages over simple area-based priors, alternative attention aggregation methods, and other head-selection strategies have not been sufficiently established. If the gains from PMass can be matched by simpler area features or alternative attention aggregation methods, its strength as a novel contribution would be weakened.
  3. On ReasonSeg test, curated training reduces the base auditor’s recall from 0.783 to 0.608 while increasing specificity from 0.514 to 0.818. This indicates that the model rejects incorrect masks more aggressively but also rejects more correct masks. Although the paper reports AUROC, it does not provide recall comparisons at matched bad-mask acceptance rates or risk–coverage curves, making it difficult to assess the benefits and costs in specific applications.
Questions for authors
  1. Clarification of the methodological contribution. What are the principal technical advances over MQA-RefAVS, beyond the change in input modalities and evaluation setting? Which challenges specific to image-based auditing motivate these advances, and which experiments directly validate them?
  2. Controlled evaluation of PMass. Does PMass retain its advantage when compared with an area-only prior and representative alternative attention aggregation or head-selection strategies within the same auditor, using identical training data and settings?
  3. Operating-point comparisons. At matched bad-mask acceptance rates, how does the curated auditor’s recall compare with the full-data auditor and the baselines? Please also report ReasonSeg recall for the attention-prior variant. Validation-selected operating points or risk–coverage curves would help establish the practical significance of the reported rejection gains. For the latter, please distinguish the error rate among accepted masks from the fraction of incorrect masks that are accepted.
Reviewer_oMgeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies reference-free verification of language-grounded segmentation masks: given an image, a referring expression and a candidate mask, predict the mask’s IoU. Simply, the auditor is Qwen2.5-VL-7B fine-tuned with LoRA to output the IoU as text. Its training data is SAM3-I masks on RefCOCO, rebalanced with synthetic near-threshold failures. An optional “attention prior” (PMass) adds to the prompt the fraction of a selected attention head’s language-to-image attention that falls inside the mask. The auditor is evaluated in-domain, on the unseen X-SAM generator, and on ReasonSeg.

Critical strengths

I couldn't find any major strengths in the paper; however here are two minor strength points:

  1. The text-swap experiment is a clean diagnostic. With the image and mask fixed, changing only the expression drops the mean predicted IoU from 0.80 to 0.08. The predictions then track the new referent’s IoU (r = 0.67), which shows the auditor uses language rather than mask appearance alone.
  2. Unlike the closest prior work, training and evaluation use real generator outputs rather than only GT-derived synthetic masks. The auditor improves threshold-free discrimination (AUROC) over the generators’ native confidence across all three settings.
Critical weaknesses
  1. Limited novelty. The paper applies the same idea as [5] (same task, an MLLM auditor fine-tuned with LoRA to predict IoU as text, trained on synthetic geometric and wrong-object errors) to images. It does not acknowledge how close it is to [5]. The port appears straightforward, with no image-specific challenge or new technique. The “independence” framing is also unconvincing: transfer across generators is expected from a large pretrained MLLM, since nothing in its input is generator-specific.
  2. Core motivation not validated. The paper is motivated by geometric vs. semantic errors, yet all results mix them. There is no breakdown by error type or IoU range. Fig. 5c suggests the auditor is unreliable on partial-overlap masks (IoU 0.1–0.5), which is the geometric-error regime it claims to address.
  3. Attention prior is not new. Grounding concentrated in a few attention heads, attention maps used as segmentation priors, and making evidence-region attention explicit are all established [1–4]. PMass applies these directly without a clear delta, and it is not normalized by mask area. [2] also shows attention can land on the correct region even when the output is wrong, which questions PMass as a correctness signal.
  4. Narrow generalization. The evidence rests on one unseen generator (X-SAM), while LISA, GLaMM, Sa2VA, and others are readily available. Results come from a single run with no confidence intervals.
  5. No downstream payoff. Despite the agent/robotics motivation, there is no best-of-N selection, no re-prompting loop, and no task success rate, whereas [5] showed downstream gains. The auditor is also never used as a GT-free reward (e.g., RL or self-training of a segmenter). That would not fix the novelty but would strengthen the contribution. The cost of rejecting correct masks is ignored: recall on the ReasonSeg test is only 0.608.
Questions for authors
  1. Beyond moving from audio-visual to images, what technical challenges did porting [5] raise, and which of your design choices address them?
  2. Can you report results separately for semantic errors (IoU < 0.1) and geometric errors (0.1–0.5)? How reliable is the auditor on partially correct masks?
  3. Have you used the auditor as a GT-free reward or for best-of-N selection, and does it improve segmentation? How do you justify rejecting about 40% of correct masks on ReasonSeg (recall 0.608)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.