- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper study “when the image and question stay fixed, how much does a textual odor cue change the judgment, relative to an irrelevant cue, and how does the model explain that change?”To this end, the paper introduces SCENTINEL, a benchmark that holds an image and question fixed while varying a textual odor description across five conditions: noodor, congruent, in-congruent, irrelevant, and adversarial. SCENTINEL implements a paired design over 1,200 items, eight visually under-specified domains, and seven VLMs.
Critical strengths
Figures and Tables are aesthetically well-presented.
Critical weaknesses
-
The evaluation metrics exhibit poor stability, and the reliability of the experimental results is questionable. As shown in Table 13, the values of the irrelevant shift rate differ by a factor of 2.35. Furthermore, in the post-fire domain, the two pipelines yield results with opposite signs (-0.144 vs. +0.370). The values presented in Tables 2 to 4 reflect not only the behavioral characteristics of the model but are also heavily influenced by the evaluation metrics, which warrants further analysis.
-
There is no matched non-odor control, so nothing in the design distinguishes sensitivity to odor descriptions from sensitivity to any appended premise.
-
The adversarial condition is confounded with two other variables: cue length and stated intensity. Section 3.3 concedes the length mismatch explicitly. The adversarial contrast of 0.300 therefore cannot be read as an effect of adversarial content.
-
The text-only control is confounded with a change of task. Interpreting the drop from 0.274 to 0.181 as evidence that "visual context also affects its magnitude" assumes the no-odor baseline distribution is comparable across profiles. Removing the image plausibly shifts the model toward uncertainty or toward the prior answer, changing both Shift(c) and Shift(irr) in a way that is not interpretable as a contrast. A stronger control would substitute an uninformative or mismatched image rather than no image, holding the response format and task constant.
-
No human reference point. The paper motivates itself through human cross-modal integration, yet without a human baseline.
-
Missing related work. The related work leans on classic VQA-prior citations from 2017-2018 plus very recent olfactory work, with a visible gap across 2024-2026 VLM evaluation.
Questions for authors
Q1: Please give the exact prompt(s) for the "smell late" profile. If a single call instructs the model to assess visually and then consider the odor, in what sense is the initial assessment independent of the cue?
Q2: What establishes that the target states are visually under-specified? Were annotators asked to answer the questions from the image alone? Given A.8's category contrasts spanning 0.014 (landfill) to 0.629 (beach), how much of the domain variation reflects differing visual determinability rather than differing cue sensitivity?
Q3: The congruent, incongruent, and irrelevant cues use "faint"/"mild" while adversarial cues use "strong". Was perceived intensity collected or balanced across conditions? If not, can the adversarial contrast (and the larger ignore-smell residual for adversarial, 0.049 versus 0.028) be separated from intensity and from the 16.57-versus-9.5-word length difference?
In addition, please refer to the Critical Weaknesses for specific comments.