acceptodds
Under review as a conference paper at ICLR 2027

BETC: Bidirectional Evidence-Target Co-Inference for Reliable Multimodal Prompt Grounding

Abstract

Multimodal prompt grounding aims to identify and localize user-specified targets in an image given natural language descriptions and visual exemplars. Existing approaches typically treat prompts as reliable instructions and directly fuse the provided information. However, due to ambiguous descriptions or incomplete observations, real-world prompts may contain both informative and misleading evidence. When different evidence sources conflict, blindly aggregating all evidence can lead to unreliable grounding. We argue that reliable grounding requires more than combining multimodal information: it requires determining which evidence truly explains the target being grounded. Based on this observation, we introduce Bidirectional Evidence–Target Co-Inference (BETC), a framework that models multimodal grounding as target-relative evidence reasoning. BETC jointly reasons over evidence and targets through a bidirectional process: evidence helps identify potential targets, while target-conditioned reasoning further determines which evidence truly supports each target. We establish two complementary protocols based on PACO and PhraseCut to evaluate grounding under partially reliable prompts, covering controlled attribute perturbations and realistic prompt mismatches. Experiments demonstrate that BETC improves robustness by dynamically adjusting evidence responsibility according to target context while maintaining competitive performance on clean inputs. Code will be released upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Learning, Vision-Language Models, Referring Expression Segmentation, Evidence Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UeLBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

BETC (Bidirectional Evidence–Target Co-Inference) addresses multimodal prompt grounding under partially reliable prompts, where textual descriptions and visual exemplars may contain conflicting or misleading information. The core observation is that existing methods treat prompts as reliable instructions and directly fuse all available evidence, which fails when cues conflict.

The framework decomposes multimodal prompts into latent evidence factors and performs bidirectional reasoning: evidence-to-target inference discovers candidate spatial hypotheses by computing response maps between evidence factors and the target image, while target-to-evidence verification evaluates which evidence factors genuinely explain each candidate through a relative gain score. A joint explanation model then infers both target belief and evidence responsibility in closed form, and the inferred responsibilities modulate the evidence representations before they are passed to the original segmentation decoder. Two auxiliary losses supervise intermediate reasoning: one encouraging evidence responses to concentrate on the ground-truth region, and one aligning candidate belief with ground-truth occupancy.

Experiments are conducted on PACO under controlled attribute corruption and on PhraseCut under natural retrieval mismatch, comparing against COSINE and several multimodal grounding baselines under both clean-only and mismatch-aware training. BETC consistently reduces degradation across all mismatch conditions while maintaining competitive clean performance. Ablations confirm that target-relative reasoning and the hard reset of non-participating evidence each contribute independently to robustness.

Critical strengths

S1: The problem setting is practically motivated and technically well-defined. Grounding under partially reliable prompts is a realistic failure mode that existing methods do not address. The paper clearly distinguishes the case where both prompts are unreliable from conventional noise settings, and the formalization of evidence responsibility as a target-conditioned relationship — rather than a fixed modality-level confidence — is conceptually clean.

S2: The joint modeling of candidate targets and evidence weights has a clear mathematical expression. The explanation model in Eq. (6) provides a principled probabilistic account of evidence participation. The closed-form derivation of target belief and evidence responsibility from the same model is a strength, and the null token design gives the model an explicit mechanism to abstain from using uninformative evidence.

S3: Ablation results support the core components. Table 3 shows a clear progression: evidence-level filtering outperforms modality-level filtering, and adding target-relative conditioning further reduces degradation from 2.50 to 1.60. The consistent gap between attention-based and intervention-based certification in the related CARE-Flow line of work is analogous to the gap shown here between target-independent and target-relative weighting, supporting the general principle. Results are reported with standard deviations over three seeds across two datasets.

Critical weaknesses

W1: The "correct target" may not be uniquely recoverable when both prompts are wrong.

The bilateral mismatch condition is the most practically important, yet it is also where the task definition is most underspecified. When both the textual description and the visual exemplar point to a different object, there may be no signal in the input from which the model can recover the original user intent. For example, if the image contains a red car and a white car, the original target is the red car, but the corrupted text describes a white car and the corrupted visual exemplar also shows a white car — the model has no basis for preferring the red car. The paper does not discuss how often this ambiguity arises in the evaluation, what BETC does in such cases, or whether the observed Bi-C performance reflects genuine recovery of the intended target or simply a bias toward the more salient object. This is a foundational issue with the task setting that deserves explicit treatment.

W2: The semantic interpretation of evidence factors remains insufficiently validated.

The evidence factors are learned latent slots extracted via cross-attention. They may encode mixed combinations of category, color, background, or texture information, and different slots may capture redundant content. As a result, a decrease in a slot's responsibility weight does not directly imply that the model has identified and suppressed a specific erroneous attribute. The qualitative responsibility traces in Figure 3 show that weights shift across conditions, but this does not confirm that the shifts correspond to semantically interpretable suppression of the corrupted attribute. More direct validation would include: controlled single-attribute modifications tracking which slot and weight change; random permutation or swapping of evidence responsibilities to verify that their specific values matter; and analysis of whether multiple slots independently encode the same information.

W3: The contribution of auxiliary supervision is not fully isolated from the core reasoning mechanism.

BETC uses ground-truth masks to supervise two intermediate objectives: concentrating evidence responses on the target region and aligning candidate belief with ground-truth occupancy. These supervision signals may themselves improve localization and robustness independently of the bidirectional reasoning mechanism. The appendix reports sensitivity to the loss weights, but the sweep does not include the case where both are set to zero. It is therefore unclear whether the performance gains come from the bidirectional evidence-target reasoning or from the additional target-localization supervision.

W4: Evaluation is limited to two datasets with attribute-level corruptions.

Both PACO and PhraseCut focus on color and material attributes. The protocol does not cover spatial relations, object identity across views, or multi-step compositional reasoning. It is unclear whether the target-relative evidence reasoning mechanism generalizes to more complex grounding scenarios where the evidence conflict involves relational or structural information rather than a single swapped attribute. The paper acknowledges this implicitly in the PhraseCut protocol description, but does not discuss it as a limitation.

Questions for authors

The two auxiliary losses supervise intermediate reasoning using ground-truth masks. Has an ablation been run without auxiliary losses while keeping all other BETC components, specifically evidence decomposition, bidirectional reasoning, and target-relative responsibility? Without this condition, the current ablation table cannot separate the contribution of the reasoning mechanism from the contribution of additional supervision.

Reviewer_ZjbTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

see weakness.

Critical strengths

see weakness.

Critical weaknesses

The paper studies an interesting problem, namely multimodal grounding when textual and visual prompts are only partially reliable, but the literature coverage, technical novelty, empirical rigor, and presentation are insufficient for ICLR.

  1. The literature review is inadequate and shows limited knowledge of the visual grounding field.
    The paper discusses a few promptable segmentation and open vocabulary works, but it omits many classic and directly relevant visual grounding studies. It also lacks discussion of robust referring expression comprehension and segmentation, so the problem positioning and the claimed novelty are difficult to assess.

  2. The technical novelty is limited and the core idea is common.
    BETC decomposes prompts into evidence slots, performs bidirectional evidence target attention, and uses latent participation to reweight prompt features before the unchanged segmentation decoder, which is close to cross attention, uncertainty weighting, mixture of experts, and hard negative mining. The probabilistic model in Eq. (6) looks principled, but the paper itself observes that the normalizer is independent of the candidate index, so the gain mainly comes from coupling evidence responsibility to candidate belief, and this coupling needs a stronger justification than soft reweighting.

  3. The empirical workload is not solid and the comparisons are not fully fair.
    The main experiments use a relatively small DINOv2 ViT S/14 and OpenCLIP ConvNeXt Base W backbone, a 437 pair PACO diagnostic subset, and limited PhraseCut test episodes, with only three seeds and no significance tests. Several baselines in Table 1 are evaluated only with clean training, while BETC is also shown with mismatch aware training, so the comparison conflates method design with training exposure and lacks parameter matched and compute matched controls.

  4. The evaluation protocol and analysis do not fully support the robustness claims.
    The construction of PACO controlled corruption and PhraseCut retrieval mismatch is mostly deferred to the appendix, and the relationship between the 437 diagnostic pairs and the full split is not clearly explained in the main paper. The paper also lacks detailed analysis of evidence slot sensitivity beyond a small sweep, failure cases under severe prompt corruption, and per bucket results, which are needed to justify the claimed generality.

  5. The presentation is poor and the manuscript feels rushed.
    The first page contains long line number blocks, the text repeats passages, equation numbering is inconsistent, and several tables are hard to parse. Key protocol details are buried in the supplement and the writing has a template generated flavor with overclaimed phrasing, which reduces readability and confidence in reproducibility.

Questions for authors

see weakness.

Reviewer_xQmXOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

BETC considers grounding with potentially unreliable text and visual prompts. It decomposes prompts into evidence factors, proposes target hypotheses, and estimates which factors support each hypothesis before passing weighted evidence to the grounding decoder. The experiments cover controlled attribute corruption on PACO and retrieval-based mismatches on PhraseCut.

Critical strengths

The candidate-discovery and target-conditioned verification split is a useful way to frame unreliable prompts. The paper tests both controlled conflicts and natural retrieval mismatches, and the fixed-evidence/changed-target analysis is a good diagnostic for the central claim.

Critical weaknesses

The paper does not report how often candidate discovery finds a region overlapping the target under prompt mismatch. Appendix G.2 distinguishes candidate coverage from candidate belief and final segmentation, but does not provide quantitative coverage results. Without these results, it is difficult to tell whether errors arise because the relevant target was not proposed or because verification assigned weight to the wrong evidence. The decoder can still recover through its image features, so missed candidates do not necessarily imply segmentation failure. Reporting performance separately for cases with and without a target-overlapping candidate would help explain when the proposed reasoning is useful. The PhraseCut results leave some uncertainty about how BETC handles misleading prompts. Clean-trained BETC scores 44.15 cIoU under bilateral mismatch in Table 2, while its No Prompt score is 43.56 in Table 8. These numbers alone do not distinguish successful suppression of misleading evidence from limited use of auxiliary prompts in difficult cases. Table 8 shows that reliable prompts can help, and the target-context analysis provides evidence that responsibilities change with context. What is still missing is a direct link between those responsibility changes and segmentation quality under mismatch. Comparing the same examples with inferred responsibilities, fixed evidence weights, and no auxiliary prompts would make that link clearer.

Questions for authors
  1. What does BETC do when it has one candidate, or when all proposed candidates are unsupported? Please evaluate these cases and clarify whether the learned NULL evidence token can reject the candidate set; Eqs. (5) and (7) otherwise assign neutral responsibility when relative gains vanish.
  2. How often does the proposal stage include a target-overlapping candidate under each mismatch condition? An oracle-candidate result would help show whether the main errors arise before verification or during evidence reasoning.
  3. On the same checkpoint and paired PhraseCut examples, how does BETC compare with No Prompt and randomly mismatched prompts? This would clarify whether the method is actively filtering misleading evidence or mostly falling back on the image pathway.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.