acceptodds
Under review as a conference paper at ICLR 2027

Scentinel: Smells Like Trouble for Vision-Language Models

Abstract

Vision-language models (VLMs) often receive auxiliary text describing conditions a camera cannot capture. Whether their use of such text reflects cue semantics, generic prompt sensitivity, or calibrated evidence weighting remains unclear. We introduce SCENTINEL, a benchmark that holds an image and question fixed while varying a textual odor description across five conditions: no odor, congruent, incongruent, irrelevant, and adversarial. Across 1,200 visually under-specified items in eight domains and seven VLMs, the original evaluation pipeline reports pooled irrelevant-controlled judgment shifts of 27.4 percentage points for incongruent cues and 30.0 points for adversarial cues. Explicit instructions to ignore odor reduce these contrasts by 89.8% and 83.7%, respectively, while delayed cues retain most of the standard-profile contrasts. Models often explicitly cite odor in their reasoning. These findings characterize judgment sensitivity across the evaluated cue pools and its substantial suppression by explicit instruction. The benchmark combines paired behavioral contrasts with an audit of stated evidence sources. Supplementary analyses document directional summaries, cue composition, classification provenance, and response validity.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
vision-language models, multimodal reasoning, cross-modal integration, auxiliary sensory evidence, olfactory cues, text priming, multimodal robustness, adversarial prompting, benchmark evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3umQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper study “when the image and question stay fixed, how much does a textual odor cue change the judgment, relative to an irrelevant cue, and how does the model explain that change?”To this end, the paper introduces SCENTINEL, a benchmark that holds an image and question fixed while varying a textual odor description across five conditions: noodor, congruent, in-congruent, irrelevant, and adversarial. SCENTINEL implements a paired design over 1,200 items, eight visually under-specified domains, and seven VLMs.

Critical strengths

Figures and Tables are aesthetically well-presented.

Critical weaknesses
  1. The evaluation metrics exhibit poor stability, and the reliability of the experimental results is questionable. As shown in Table 13, the values of the irrelevant shift rate differ by a factor of 2.35. Furthermore, in the post-fire domain, the two pipelines yield results with opposite signs (-0.144 vs. +0.370). The values presented in Tables 2 to 4 reflect not only the behavioral characteristics of the model but are also heavily influenced by the evaluation metrics, which warrants further analysis.

  2. There is no matched non-odor control, so nothing in the design distinguishes sensitivity to odor descriptions from sensitivity to any appended premise.

  3. The adversarial condition is confounded with two other variables: cue length and stated intensity. Section 3.3 concedes the length mismatch explicitly. The adversarial contrast of 0.300 therefore cannot be read as an effect of adversarial content.

  4. The text-only control is confounded with a change of task. Interpreting the drop from 0.274 to 0.181 as evidence that "visual context also affects its magnitude" assumes the no-odor baseline distribution is comparable across profiles. Removing the image plausibly shifts the model toward uncertainty or toward the prior answer, changing both Shift(c) and Shift(irr) in a way that is not interpretable as a contrast. A stronger control would substitute an uninformative or mismatched image rather than no image, holding the response format and task constant.

  5. No human reference point. The paper motivates itself through human cross-modal integration, yet without a human baseline.

  6. Missing related work. The related work leans on classic VQA-prior citations from 2017-2018 plus very recent olfactory work, with a visible gap across 2024-2026 VLM evaluation.

Questions for authors

Q1: Please give the exact prompt(s) for the "smell late" profile. If a single call instructs the model to assess visually and then consider the odor, in what sense is the initial assessment independent of the cue?

Q2: What establishes that the target states are visually under-specified? Were annotators asked to answer the questions from the image alone? Given A.8's category contrasts spanning 0.014 (landfill) to 0.629 (beach), how much of the domain variation reflects differing visual determinability rather than differing cue sensitivity?

Q3: The congruent, incongruent, and irrelevant cues use "faint"/"mild" while adversarial cues use "strong". Was perceived intensity collected or balanced across conditions? If not, can the adversarial contrast (and the larger ignore-smell residual for adversarial, 0.049 versus 0.028) be separated from intensity and from the 16.57-versus-9.5-word length difference?

In addition, please refer to the Critical Weaknesses for specific comments.

Reviewer_6HwWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces SCENTINEL, a benchmark for how VLMs use auxiliary text about properties a camera cannot capture, with odor as the test case. Each of 1,200 image–question pairs (8 domains × 150 items; images from Food101 and Places365) asks about a visually under-specified hidden state, such as food freshness or a gas hazard. The image and question are held fixed while the odor cue varies across five conditions: none, congruent, incongruent, irrelevant (the control), and adversarial. The main metric is the shift in judgment relative to no odor, minus the shift under the irrelevant cue. Three control profiles remove the image (text only), instruct the model to ignore odor (ignore smell), or supply the cue after an initial judgment (smell late). Structured outputs separate visible evidence from rationale, which allows two audits: whether the model attributes its change to odor, and whether odor “leaks” into the visible-evidence field. Seven models are evaluated, with 42,000 standard responses plus three ablation profiles. Pooled contrasts: incongruent +27.4 pp, adversarial +30.0 pp, congruent +2.3 pp; text only reduces the contrasts by 34–45%; ignore smell reduces them by 84–90%; smell late retains about 90%; 86% of shifted responses cite odor, and leakage into the visible field is 0.1–2.2%.

Critical strengths

The paired design is simple and sensible. Fixing the image and question and varying only the odor text, with an irrelevant-cue control, isolates the effect of the cue better than most prompt-sensitivity studies do. The three control profiles add real information. The ignore-smell result in particular (contrasts drop to about 3–5 pp) shows that models mostly comply when told to restrict their evidence. Separating a visible-evidence field from the rationale is a nice, cheap way to check whether models present the smell as something they can see, and leakage turns out to be rare. The evaluation covers seven current models across eight domains.

Critical weaknesses
  1. The benchmark has no way to say whether a shift is wrong. The latent state is never verified, and every question is one where odor is plausibly relevant. Becoming more cautious about food safety after being told it smells sour may be the right call. There are also no odor-irrelevant questions or images where the answer is visually obvious, so the paper never shows a clear failure. The “vulnerability” framing goes beyond what the results support.

  2. The control condition is not neutral. The irrelevant cue alone changes 37% of judgments, and there is no repeatability baseline (the same input run twice) to show how much of this is ordinary run-to-run noise. Since every reported contrast is measured against this pool of only 15 cues, this matters for all the main numbers.

  3. The main classification pipeline is never described or validated. The auxiliary classifier disagrees with it substantially, including an opposite sign for fire aftermath (−0.14 vs. +0.37), and no accuracy against human labels is reported.

  4. The confidence intervals are too narrow. Cue strings are reused hundreds of times, and the authors’ own cue-level resampling (Appendix A.5) gives intervals 5–7× wider than those in the main tables. This and other important qualifiers, such as GLM’s 37–59% parse failures, appear only in the appendix.

Questions for authors

here are some simple and easy to add improvement that can help:

  1. Validate the classifier. All main results depend on the “original classification pipeline,” but the paper never describes it. Please explain how answers are labeled and report its accuracy on a few hundred hand-labeled responses, especially for fire aftermath, where the two classifiers disagree in sign.

  2. Measure run-to-run noise. Send a subset of identical inputs twice and report how often the judgment changes. This shows how much of the 37% irrelevant-cue shift is ordinary sampling variation rather than an effect of the cue.

  3. Add questions where odor should not matter. Ask about the same images with questions odor cannot affect, such as “How many plates are on the table?” for a Food101 image, under the same five cue conditions. Any shift there is clearly inappropriate, which gives the benchmark a case where a change is unambiguously wrong.

  4. Report realistic uncertainty and sample sizes. Each cue string is reused across hundreds of items, and resampling by cue (Appendix A.5) gives intervals 5–7 times wider than those in Table 2. Please use these wider intervals for the headline results, and add valid response counts per model, since some models (e.g., GLM) lose up to 59% of responses to parse failures.

  5. Use the data already collected. The outputs already include judgment direction and confidence. Reporting whether incongruent cues push answers toward caution, and whether confidence changes when judgments shift, would make the main contrasts much easier to interpret.

Reviewer_4Q1DOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces SCENTINEL, which is used to evaluate how vision-language models make use of auxiliary smell text that describes properties of the depicted scenes that cannot be reliably determined from images alone. The benchmark includes 1,200 image–question pairs across eight domains, with each pair tested under five conditions: no odor, congruent, incongruent, irrelevant, and adversarial. The authors evaluate seven models and analyze the effects of visual inputs, evidence-use instructions, and information timing through three control settings: text only, ignore smell, and smell later. The study found that adversarial smell text can significantly alter model predictions.

Critical strengths

1.This paired comparison experiment setup is a practical troubleshooting approach. The experiment keeps the images and questions fixed and only changes the additional descriptive text, making it easy to directly compare the impact of different descriptions. Especially, the fact that the effect drops significantly after the model receives instructions, and that delayed description display can still change judgments, is very useful for understanding how the model references information and adjusts its own judgments.

2.By separating the visual-evidence field from the rationale, this paper can distinguish between two situations: whether the model admits it referenced the extra text, or falsely claims that the content in the text was seen in the image. This approach can also be used to check various types of supplemental information.

Critical weaknesses

1.The number, distribution, and statistical handling of cues limit fine-grained conclusions. Out of 1,200 samples, only 127 unique smell strings were used, with only 15 unique irrelevant cue strings, and the domain distribution was uneven; resampling based on samples didn't fully capture the uncertainty in cue generalization. Adversarial descriptions are longer, and some models had high rates of unparseable structured outputs, which further limits the reliability of domain comparisons and model ranking.

2.The evaluation measured changes in judgment but didn’t fully assess whether the use of evidence was appropriate. In tasks where visual information is limited, adjusting judgments based on smell reports might actually be reasonable; lacking control for the true state and source reliability means the change rate can’t distinguish between reasonable updates and over-reliance. Although the paper acknowledges this limitation, it restricts the interpretation of the observed shifts as evidence of vulnerability or inappropriate reliance.

Questions for authors

1.What is the direction of the change? The 27.4 percentage points measure category changes without direction. Could the authors report, according to A.13, excess-caution transition rates, net differences in direction, and complete transition matrices for incongruent and adversarial conditions, to distinguish between 'easier to change judgments' and 'easier to become cautious'?

2.Does the extra effect in adversarial conditions come from description length? Can the 2.6 percentage point difference between adversarial and incongruent conditions be tested through length-matched or length-controlled analysis? If existing cue lengths don't overlap, please clarify whether the current data can distinguish the effects of length versus content.

3.How should the asymmetry between benign and negative cues be explained? Benign cues only increase changes by 2.3 percentage points relative to the irrelevant condition—could this be due to the initial judgment distribution and category thresholds? Could the authors test this explanation by considering baseline categories, transition direction, and confidence changes?

4.What evidence could support explanations specific to smell or unreasonable reliance? Are there already matched non-smell controls or source reliability experiments? If not, please clarify whether the current design can distinguish odor-specific effects from general sensitivity to auxiliary text and explain how to distinguish reasonable updating from over-reliance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.