acceptodds
Under review as a conference paper at ICLR 2027

CuE: A Cue Engineering Framework for Diagnosing and Mitigating Shortcut Effects on Zero-shot MLLM Recognition

Abstract

Generative multimodal large language models (MLLMs) offer open-vocabulary recognition, but their decisions can depend on spurious visual or linguistic cues. We introduce CuE, a training-free framework that organizes candidate cue mechanisms, constructs target-conditioned cue hypotheses with an offline helper LLM, and performs fixed-label diagnosis before cue-matched remediation on frozen MLLMs based on a taxonomy of cue families introduced in this study. On a stratified 12-target COCO panel, we evaluate four target MLLMs from three model families. Visual diagnosis reveals recurring sensitivities: target recall is %–% lower in competitor-present groups, and partial-evidence groups expose %–% recall gaps. Language-side diagnosis further shows model-dependent sensitivities: query rewrite flips %–% of decisions, while language-prior probes and text interference increase miss rates by %–% and %–%, respectively. Cue-matched remediation improves balanced accuracy consistently across the evaluated visual settings, whereas generic debiasing baselines can over-constrain recognition and reduce recall. These findings suggest that robust zero-shot MLLM recognition requires diagnosing the relevant shortcut cue families before applying remediation, rather than relying on one-size-fits-all prompting fixes.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
spurious correlations, MLLM, shortcut, zero-shot recognition

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nAbmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces CuE, a training-free framework designed to diagnose and mitigate shortcut cue reliance in zero-shot generative Multimodal Large Language Models. The authors formalize shortcut sensitivity by defining an extensible taxonomy encompassing four visual cue families and three linguistic cue families. An offline helper LLM (Qwen3.5-9B) generates and filters candidate cue hypotheses for queried target concepts. The authors evaluate CuE on a stratified panel of 12 COCO target categories using four MLLMs. They report substantial cue-dependent recall and specificity gaps for competitor, context, and partial-evidence conditions, as well as model-dependent sensitivity to query rewrites, language priors, and text interference. For visual cues, cue-matched prompting improves balanced accuracy in all 12 reported model–cue settings and is more consistent than three generic prompting baselines.

Critical strengths
  1. A unified and interpretable diagnosis framework. CuE connects cue hypothesis generation, fixed-label grouped evaluation, and cue-matched remediation in a coherent pipeline. Compared with reporting aggregate recognition accuracy alone, the proposed evaluation encourages inspection of recall and specificity under structured conditions. This is practically useful for exposing systematic failure patterns that may be hidden by high average accuracy.

  2. Evaluation across multiple model families and both modalities. The paper evaluates four frozen MLLMs from different families and considers both visual and linguistic sources of brittleness. In particular, the strong variation across models and text-interference subtypes supports the paper’s motivation that a single generic debiasing prompt may not be appropriate for every model or failure mode.

Critical weaknesses
  1. Limited Scale and Narrow Domain Coverage.

The empirical validation is restricted to a 12-target panel derived from MS-COCO. Furthermore, using the COCO train split rather than standard validation/test sets or out-of-domain benchmarks raises questions regarding whether the observed behaviors generalize to more diverse, fine-grained, or complex real-world open-vocabulary scenarios. And many gains in Table 1 are only 1–2 percentage points, but the paper provides no confidence intervals, paired tests, or per-target distributions.

  1. The grouped visual evaluation does not support causal attribution to the designated cue.

The conditional differences in Equations (1)–(2) are not controlled interventions. Cue-present and cue-absent groups consist of different natural images, and fixing the target label does not control other visual factors. For example, target-positive images containing a competitor may also contain smaller, more occluded, or more crowded target instances. Likewise, images lacking typical partial evidence may involve unusual or intrinsically difficult viewpoints. Therefore, the recall gaps may reflect general sample difficulty rather than competitive suppression or shortcut reliance.

Questions for authors
  1. Open-World / Unknown-Cue Deployment: How does CuE function when the image-specific cue condition is unknown at inference time? If all cue-remediation instructions are combined into a single unified prompt, does the model retain its balanced accuracy improvements, or does it suffer from prompt-induced over-constraining similar to the generic debiasing baselines?
2 comments

We highly appreciate your valuable time in reviewing this work and your critical comments.

Weakness 1: Scale, COCO train, generalization, and uncertainty

The 12 targets were deliberately selected across four semantic groups and evaluated on four MLLMs from three model families. This design tests whether the diagnostic patterns and remediation benefits recur across categories and architectures, providing evidence for the framework beyond individual examples. We acknowledge that 12 categories do not cover the diversity of open-vocabulary recognition. We will extend the evaluation to broader categories and domains, as well as reporting per-target results with standard deviations.

COCO train provides the annotated pool needed to construct four balanced target/cue groups, each containing 500 images per target and setting. No target-model training or parameter updating is performed. COCO test annotations are not publicly released for constructing these groups and computing our conditional metrics, while the 5,000-image validation split spans 80 categories and offers limited samples for some target/cue combinations. These requirements motivate using the larger annotated training split for evaluation.

A central goal of spurious-correlation research is to improve performance on difficult groups where familiar cues are absent or misleading, despite high average accuracy. These groups—such as context-absent positives and competitor-present negatives—are precisely what our remediation targets. For example, Gemma-4 competitor prompting improves specificity on competitor-present negatives by 7.5 points (Table 1). The corresponding BAcc gain is 2.65 points because it averages this improvement with changes in the other groups. Thus, modest aggregate gains can obscure substantial improvements on the failure groups that motivate spurious-correlation research.

Weakness 2: The visual gaps do not establish causality

Lower recall could potentially caused by smaller or more occluded targets. However, the observed pattern also includes false positives: across all four models, specificity falls by 5.0–14.3 points on competitor-present negatives and 10.3–17.7 points on partial-evidence-present negatives (Table 3). Because the target is absent, reduced target visibility (smaller, occluded, or crowded) cannot explain this component of the results. For partial evidence, the diagnostic concern is falsely recognizing an absent target, rather than the legitimate benefit of visible parts when the target is present.

The remediation experiment provides complementary evidenced: base and cue-matched prompts are evaluated on the same images (section 3.5), so their performance difference cannot arise from giving the remedy easier examples. This supports the intervention's practical benefit.

Table 5 reports target-label agreement of 98.3%, 95.0%, and 98.3% for context, partial evidence, and competitors. This supports annotation reliability, but difficulty confounds remain (as acknowleded in Appendix C). We therefore interpret the gaps as cue-associated failure patterns rather than causal estimates. In future research, we will control for target size, occlusion, viewpoint, and scene complexity through matched or stratified analyses to strengthen causal attribution of the diagnostic gaps.

Question: Open-World / Unknown-Cue Deployment

CuE's contribution includes actionable empirical insights into recognition failures across cue families and models. Its failure profiles can inform which counterexamples, augmentations, or concept distinctions developers should emphasize during subsequent model training and evaluation. Separating failure families also helps researchers design interventions for particular errors and understand where a generic remedy falls short. The value of this diagnosis therefore extends beyond the immediate prompt-based remedy.

For users and practitioners using MLLMs for recognition or visual judging, CuE also offers prompt-based intervention when modifying the model is unavailable or inconvenient. Diagnosis takes place offline; subsequent predictions use the chosen instruction in a single model call (section 3.5).

The prompt does not require the incoming image's cue label. It names a possible confusion without asserting that the cue is present, and the same remedy is evaluated on both cue-present and cue-absent images. The family-wise design first establishes how each instruction affects its corresponding failure pattern. Table 9 shows that stronger instructions can hurt recognition, so gains from separate remedies should not be assumed to add up. However, as noted in Appendix C, evaluating combined remedies is our research next step.

Thank you for your response. After considering the response, I will maintain my score.

Reviewer_a3a3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The manuscript introduces CuE, a training-free cue engineering framework for diagnosing and remediating recognition failures induced by shortcut cues in frozen zero-shot MLLMs. CuE organizes candidate conditions into seven cue families, uses an offline helper LLM to construct target-conditioned cue hypotheses, performs controlled visual and linguistic diagnostics under fixed ground-truth target labels, produces a failure profile, and then applies cue-matched remediation through family-specific cue-conditioned prompts. The main contributions are: (1) formalizing cue-driven shortcut sensitivity for zero-shot MLLM recognition under a fixed ground-truth target; (2) an extensible cue-family taxonomy and helper-LLM-based target-conditioned cue library; (3) controlled visual and linguistic diagnostics that isolate cue conditions; and (4) cue-matched remediation without target-dataset fine-tuning or auxiliary modules. Validation uses a stratified 12-target COCO panel with four MLLMs. The diagnosis reports recurring sensitivities to competitor, context, and partial-evidence cues, as well as model-dependent linguistic sensitivities. Cue-matched remediation improves balanced accuracy in all 12 evaluated model-cue settings, with smaller recall-specificity trade-offs than taxonomy-agnostic prompting baselines.

Critical strengths

1.By separating cue-family diagnosis from family-specific cue-matched prompts, CuE turns a failure profile into an actionable intervention rather than applying a one-size-fits-all prompt. This is consequential because it provides a black-box, frozen-MLLM procedure that connects a candidate cue family to observed recognition failures and to a targeted remedy. 2.The reported patterns—competitor co-presence suppressing target recognition, partial evidence producing strong recall asymmetry and specificity loss, and text interference being model-dependent—are useful for understanding how shortcut cues affect zero-shot recognition. The consistent 12/12 improvement over taxonomy-agnostic prompting, although modest, supports the value of cue matching.

Critical weaknesses

1.Cue-present and cue-absent groups are constructed from naturally occurring images without matching object size, occlusion, viewpoint, salience, or scene complexity. The cue-image audit quantifies label noise but does not remove recognition-difficulty confounds. Therefore, the reported Recall_gap and Spec_gap may reflect general recognition difficulty rather than model sensitivity to the named cue family. This directly weakens both the diagnosis and the rationale for cue-matched remediation. 2.The study uses 12 COCO targets, four MLLMs, and binary yes/no recognition. Remediation gains are 0.73–2.65 balanced-accuracy points, with no confidence intervals, seed variance, or per-target significance analysis. Comparisons are limited to taxonomy-agnostic prompting baselines; the manuscript does not establish that the target-conditioned cue library and cue-matched prompts are more effective than other applicable shortcut-diagnosis or mitigation procedures for frozen MLLMs. Thus, the claim of more reliable zero-shot recognition and systematic shortcut-cue discovery is not fully substantiated.

Questions for authors

1.Can you add matched-subset or stratified analyses controlling for object size, occlusion, viewpoint, salience, and scene complexity? How much do Recall_gap and Spec_gap change, and do cue-matched remediation gains persist under these controls? 2.Can you report per-target results, confidence intervals or statistical tests, and decoding settings/seeds? Are the 12/12 balanced-accuracy improvements statistically significant and consistent across targets? 3.How sensitive are downstream diagnosis and cue-matched remediation to the helper LLM? Table 6 reports upstream cross-filter agreement, but can you rerun diagnosis and remediation with libraries constructed by different helper LLMs? 4.Can you clarify how an image affected by multiple cue families would be handled? Is composition of cue-conditioned prompts evaluated, or is this left entirely to future work? 5.Can you justify the omission of comparisons to shortcut-diagnosis or mitigation procedures that are applicable to frozen MLLMs, beyond the taxonomy-agnostic prompting baselines?

2 comments

Thank you for recognizing the value of the paper. We clarify the evidence and scope below.

Weakness 1 & Question 1: Image difficulty may explain the visual gaps

Two comparisons need to be distinguished. The diagnostic comparison uses different natural images to locate conditional failure patterns. The remediation comparison changes the prompt on the same images (§3.5). Size, occlusion, and scene complexity can potentially confound the first comparison, but a difference in image difficulty between the base and remedy inputs cannot account for the measured gains in the second.

Furthermore, our diagnosis does not rely solely on lower recall in potentially harder images. For example, across all four models, specificity drops by 5.0–14.3 points on competitor-present negatives and 10.3–17.7 points on partial-evidence-present negatives (Table 3). Since the target is absent, these false positives cannot be explained by smaller or more occluded target instances.

However, other factors like viewpoint and scene complexity may still influence the gaps, so we interpret them as cue-associated failure patterns rather than causal estimates, as presented in Appendix C. Future work will control these factors through matched or stratified analyses.

Weakness 2 & Question 2: Scope, small gains, uncertainty

CuE contributes a systematic way to investigate recognition failures and empirical insights into their variation across cues and models. The 12 targets span four semantic groups and the four MLLMs span three families, testing recurrence across categories and architectures. Binary recognition allows us to measure missed detections and false positives separately. The resulting profiles can guide counterexample collection, augmentation, concept distinctions in subsequent training and evaluation, and interventions for particular errors. We acknowledge the limited vocabulary, and will extend evaluation to more categories and domains. Table 7 reports standard deviations; we will clarify the replication details and provide per-target results and paired uncertainty estimates.

Improving difficult groups where cues are absent or misleading is a central objective of spurious-correlation research. That is where the remediation should be assessed. For example, Gemma-4 gains 7.5 specificity points on competitor-present negatives, while its BAcc gain seems small (2.65 points) because the aggregate averages changes across all four groups (Table 1). Table 2 also shows positive average gains in all 12 model–family settings, versus 2, 5, and 2 for the three generic strategies.

Weakness 2 & Question 5: comparisons

CuE contributes a systematic analysis of recognition failures. For remediation, we compare interventions under the same practical access constraint: users can submit images and text and obtain generated answers. This accommodates users who cannot access model internals or prefer to avoid model modification. The three baselines are adapted from MM-SpuBench [1] and evaluated on the same models and images. They test whether general reasoning or shortcut warnings provide the same benefits as cue-specific guidance. Other methods freeze model weights but require additional access. VCD [2] and M3ID [3] combine token scores from multiple conditioning branches; PAI [4] modifies attention; and ICD’s reference implementation [5] perturbs instructions within an instruction-aware Q-Former. The inference-only MID variant [6] likewise requires prediction scores and adapted counterfactual inputs. These methods are relevant comparisons when such access is available, but cannot be applied through an interface that returns only generated answers. Our comparative claim therefore concerns the tested prompt interventions.

[1] Wenqian Ye et al. MM-SpuBench: Towards Better Understanding of Spurious Biases in Multimodal LLMs. KDD, 2026.

[2] Sicong Leng et al. Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding. CVPR, 2024. (VCD)

[3] Alessandro Favero et al. Multi-Modal Hallucination Control by Visual Information Grounding. CVPR, 2024. (M3ID)

[4] Shi Liu, Kecheng Zheng, and Wei Chen. Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs. ECCV, 2024. (PAI)

[5] Xintong Wang et al. Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding. Findings of ACL, 2024. (ICD)

[6] Zichen Wu, Hsiu-Yuan Huang, and Yunfang Wu. Beyond Spurious Signals: Debiasing Multimodal Large Language Models via Counterfactual Inference and Adaptive Expert Routing. Findings of EMNLP, 2025. (MID, §4.1)

Question 3: Does changing the helper change downstream results

The helper generates hypotheses offline; target-model predictions determine the failure profiles. Libraries are filtered, audited, and reused across models (§3.3). Table 6 reports 86.9%–88.4% cross-filter agreement and 84.6%–86.1% accepted-set overlap, indicating that filtering judgments are mainly consistent across helpers. Together with the human audit, this supports the reliability of library construction.

Extending this assessment downstream involves 4 helpers × 4 downstream models × 4 prompts × 3 settings × 12 classes × 2,000 images × 3 seeds = 13,824,000 image–prompt inference call. This workload exceeds what we can complete within the rebuttal period. We will investigate downstream sensitivity in stages, starting with alternative helper-conditioned prompts on common images.

Question 4: How are simultaneous cues handled?

We study families separately to understand which instruction addresses which error. This separation also can help researchers design targeted interventions and identify where generic remedies fail. It does not assume that natural images contain only one kind of cue. Within each family, the same remedy is applied to cue-present and cue-absent images without supplying their cue labels.

Prompt composition and automatic selection among families have not been evaluated. Table 9 shows that hard competitor prompting reduces Qwen2.5 BAcc from 90.1% to 85.4%, whereas medium prompting reaches 91.1%. This strength ablation warns against assuming that more instructions yield larger gains; it does not measure composition. We will investigate combined prompts on mixed-cue images, as outlined in Appendix C.

Reviewer_UQY3OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CuE, a training-free framework for improving MLLM zero-shot visual recognition by diagnosing model-specific sensitivities to spurious cues and mitigating them through cue-specific prompting. The framework organizes potential cues into a taxonomy of cue families, uses a helper LLM to generate target-conditioned cue hypotheses, and conducts controlled evaluations to identify cue-related failure patterns. Based on the diagnosis, it applies cue-specific prompts to mitigate the corresponding sensitivities.

Experiments on four MLLMs and 12 object categories show that MLLMs exhibit sensitivity to different types of cues despite achieving high overall recognition accuracy. The results also demonstrate that cue-matched prompting yields more consistent performance improvements than generic prompting baselines.

Critical strengths
  • The paper introduces a systematic taxonomy of shortcut cues and leverages a helper LLM to automatically generate target-conditioned cue hypotheses, reducing the need for manually specified or annotated spurious correlations.

  • Rather than treating data-level correlations as direct evidence of shortcut reliance, the paper attempts to identify whether MLLMs are actually sensitive to these cues through targeted diagnostic evaluations.

Critical weaknesses
  • The evaluation of the proposed diagnosis-guided remediation framework is incomplete in several important aspects:

    1. The remediation experiments appear to apply predefined cue-matched prompts directly, rather than selecting whether and how to perform remediation based on the diagnosis results. Therefore, it is unclear whether the diagnosis stage actually contributes to the effectiveness of remediation.

    2. The experiments only evaluate cue-matched remediation on the corresponding cue-specific evaluation sets, without examining how a remediation prompt affects performance under other cue conditions. This also raises concerns about the fairness of the comparison with generic prompting baselines, as CuE uses prompts tailored to each evaluation setting while the baselines use fixed prompts.

    3. In realistic scenarios, an image may involve multiple types of shortcut cues simultaneously. However, the paper does not evaluate the effectiveness of combining multiple remediation strategies.

    4. Although the framework includes three linguistic cue families and evaluates their effects during diagnosis, the remediation experiments only cover visual cues, leaving the effectiveness of linguistic cue remediation unexplored.

Questions for authors
  • Could the authors clarify the intended real-world deployment scenario of CuE? Is it designed as a dedicated recognition system with predefined targets and offline diagnosis, or as a general-purpose MLLM that supports open-ended user interactions? The latter seems particularly relevant given the evaluation of linguistic cues such as user judgments and text interference.
2 comments

We appreciate your great efforts in reviewing this work and valuable comments offered. We would like to address your concerns via our following responses.

Weakness 1: What does diagnosis add to remediation?

The predefined templates specify possible actions; diagnosis supplies the model's failure profile and identifies what those actions should address. Confusing a target with a competitor, missing it without familiar context, and mistaking partial evidence for the whole object suggest different corrective instructions. CuE makes these differences visible through separate recall and specificity measurements instead of assuming that every model needs the same generic warning.

The intervention results align with those profiles. For Qwen3, competitor-aware prompting improves recall and specificity in competitor-present images by 2.7 and 4.4 percentage points, while the corresponding cue-absent changes are +0.1 and -0.2 points (Table 1). Partial-evidence prompting improves specificity on cue-present negatives by 5.5 points. Across the 12 model–family settings, CuE produces positive average balanced-accuracy changes in all 12, compared with 2, 5, and 2 settings for the three generic strategies (Table 2). This supports diagnosis as an interpretable basis for choosing the failure to address.

Weakness 2: Evaluation only on the corresponding cue-specific evaluation sets

This evaluation design aligns with one of our major discovery: Cue matching remediation outperforms taxonomy-agnostic prompting and the results of CuE are more consistent. However, the cue-matched prompts use target-conditioned cue hypotheses generated by helper LLMs; it receives neither the image's true target label nor its true cue-presence label. The reported gains therefore do not depend on applying the instruction only to images known to contain the cue.

The comparison with generic prompting baselines asks whether explicit cue information improves on general prompting under the same access constraints. CuE is intended for common users who can submit an image and a prompt but cannot access model weights, hidden representations, attention maps, or full token scores. The three baselines use that same interface and are adapted from MM-SpuBench [1]. CuE's cue information is the intervention being evaluated.

[1] Wenqian Ye et al. MM-SpuBench: Towards Better Understanding of Spurious Biases in Multimodal LLMs. KDD, 2026.

Weakness 3: Combining remedies when several cues occur together

The family-wise design lets us inspect which instruction changes which failure pattern. Combining all instructions from the beginning would make their individual contributions harder to assess. However, appendix C also notices this limit, and we acknowledge that evaluating composition of remediation strategies is one of our next research questions after establishing the contributions in this paper.

Weakness 4: Missing linguistic remediation

We apologize for the unclear presentation. In dedicated visual recognition, linuigstic conditions such as extraneous user judgments and unrelated claims can be omitted while preserving the target question. Our linguistic evaluations support this practice: contradictory facts increase misses by 42.4 points for Qwen3 and 5.6 points for Gemma-4 relative to the clean query (Figure 3). Removing the added text restores the base-query condition already evaluated.

Question: What is the intended deployment setting

CuE serves both evaluation and deployment. Researchers can use it to systematically assess recognition sensitivity to candidate spurious cues and identify model-specific failures hidden by aggregate accuracy. Users, developers, and those using MLLMs as visual judges, can use cue-hypothesis generation and cue-matched prompting without accessing weights or internal representations to avoid the effort of model modification and to improve recognition robustness to potential spurious correlations. Linguistic probes remain relevant because paraphrases, user judgments, and distracting text can affect recognition even in dedicated applications. Our experiments validate family-wise visual remediation; automatic selection and unrestricted conversation remain untested.

Thank you for the detailed responses and clarifications.

I appreciate the authors' explanation of the cue-matched evaluation setting and the role of offline diagnosis. I agree that cue-matched evaluation is reasonable for assessing the effectiveness of individual remediation strategies, and that offline diagnosis followed by fixed remediation can be a practical deployment approach.

However, I remain concerned about the completeness of the proposed remediation framework. As a model may exhibit sensitivity to multiple cue families for the same target, a practical deployment would need to determine how to handle multiple diagnosed failure modes. There are two possible approaches:

  • Combining multiple remedies: If multiple cue-specific prompts are combined to address all diagnosed sensitivities, their interactions and combined effectiveness need to be evaluated. The current experiments do not examine such combinations.
  • Selecting a single remedy: If only one remediation strategy is selected despite the presence of multiple diagnosed sensitivities, cross-cue effects inevitably become relevant, as the selected remedy will also be applied to images affected by other cue families. Therefore, it is necessary to evaluate whether this remedy negatively affects performance under those other cue conditions. However, the current experiments only evaluate each remediation on its corresponding cue-specific evaluation set, leaving these cross-cue effects unexplored.

Therefore, neither approach is sufficiently evaluated to establish how CuE would handle multiple diagnosed sensitivities in practice. While the current results demonstrate the benefits of individual cue-matched interventions, I believe this limitation still leaves the overall remediation framework insufficiently validated.

Nevertheless, the authors' clarifications have addressed some of my initial concerns. Considering these clarifications and the remaining limitations, I have decided to raise my rating to Weak Rejection (2), which I believe best reflects my current assessment of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.