- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper asks a simple but important question: are generative vision-language models (VLMs) bad at zero-shot image classification because their visual understanding is weak, or because asking them to generate a label one token at a time is the wrong way to get a classification answer out of them? The authors argue that much of the gap lies in this inference interface, not in the model's representations. Their solution is to sidestep generation altogether. Instead of prompting the model to output a class name, they extract its internal hidden states at the moment it is "about to" name the object — once from an image prompt, and once for each candidate class described in text — and then classify by simple similarity matching. A second component, Semantic Equivalence Rectification (SER), enriches each class with LLM-generated aliases, scientific names, and descriptions, and only activates this expanded vocabulary when the model is uncertain. The approach is tested on eight benchmarks and seven generative VLMs. Latent matching consistently outperforms a reproduced closed-world prompting baseline.
Critical strengths
- The distinction between representational capacity and inference-interface limitations is important. As generative VLMs are increasingly deployed as general-purpose multimodal systems, enabling reliable fixed-output classification without fine-tuning or an additional contrastive model has clear practical value.
- The proposed latent-matching procedure is architecture-agnostic and requires no parameter updates. It could plausibly be applied to a range of instruction-tuned VLMs without retraining or benchmark-specific adaptation.
- The paper evaluates seven generative VLMs from multiple model families on eight benchmarks, including fine-grained recognition, general object classification, food, textures, etc. The consistency of the improvement across most settings strengthens the empirical benefits.
- The investigation of Taxonomic Bias is one of the paper’s more original contributions. The wrong-class mapping experiment is a meaningful attempt to determine whether SER is genuinely exploiting taxonomy–class correspondence rather than simply adding more text.
Critical weaknesses
-
The reproduced closed-world baseline requires the model to generate one label and then parses the response. This does not exhaust the reasonable ways to perform classification with a generative VLM. Stronger and more direct baselines should include, at minimum:
- ranking candidates by their token or sequence probabilities rather than relying on a generated answer (the "Prob Inference" variants in Zhang et al. 2024);
- constrained decoding restricted to candidate labels;
- multiple prompt templates and answer-format instructions;
- comparison with recent probability-based or calibrated classification protocols for generative VLMs, e.g., [1][2][3]
The extremely low baseline scores for some model–dataset pairs—for example, approximately 0% or 0.05% on Cars196—suggest that prompt construction, candidate enumeration, response formatting, or parsing may be responsible for a substantial part of the observed gap.
-
SER uses an external LLM to generate class-specific scientific names, aliases, synonyms, and descriptions. Although no target images or image–label pairs are used, this is still an inference-time semantic prior derived from external knowledge. It is not directly comparable to a method using only benchmark class names, [4] and [5] are comparable semantic augmentation works. The paper correctly reports “Ours” and “+SER” separately, but the abstract and some conclusions still present SER as part of the general framework without consistently emphasizing this additional knowledge source. Moreover, the baseline should also be evaluated with the same synonym-aware answer mapping; otherwise, SER receives an advantage unavailable to the generative baseline.
-
Max pooling over externally generated variants can favor classes with more expressive, generic, or numerically extensive descriptions. The audit itself shows that unrelated broad expansions recover 59 of 113 corrected samples, which weakens the interpretation of SER as purely taxonomy-aware rectification. The wrong-class mapping experiment is useful, but it does not fully separate taxonomy alignment from prototype geometry and generic description effects.
-
The geometric argument establishes only that same-token contexts face related constraints in the space visible to the LM head; it does not imply cosine proximity. The controlled probe is text-only and therefore does not directly validate cross-modal visual–textual alignment. The main classification results show task-level utility, but they do not isolate whether matching works because of shared semantics, shared prompt syntax, modality-specific artifacts, or spurious correlations.
References
[1] Multimodal Large Language Models as Image Classifiers, ICLR 2026.
[2] On Large Multimodal Models as Open-World Image Classifiers, ICCV 2025.
[3] Revisiting MLLMs: An In-Depth Analysis of Image Classification Abilities, ArXiv 2024.
[4] Visual Classification via Description from Large Language Models, ICLR 2023.
[5] What does a platypus look like? Generating customized prompts for zero-shot image classification, ICCV 2023.
Questions for authors
- Did the authors evaluate the strongest inference protocol available on their own reproduced baseline: scoring each candidate label by its conditional token probability and taking the argmax (the "Prob Inference" variants evaluated in Zhang et al. 2024)?
- What happens when the same Gemini-generated aliases, scientific names, and descriptions are made available to the closed-world baseline — e.g., as answer-normalization vocabulary, or as additional enumerated options?
- The reproduced baseline scores 0.00% (LLaVA-NeXT, Cars196) and 0.05% (LLaVA-v1.5, Cars196). For these near-zero cases, what fraction of outputs were successfully parsed, and did the authors manually inspect a sample of failures?
- How many tokens does the 1,000-class closed-world prompt occupy for each model, and did every backbone process it without context truncation or degraded attention?
- Is the number of variants M identical for every class within a dataset, and are the variants' lengths roughly comparable?
- The paper invokes permutation-order sensitivity of enumerated lists as motivation, yet reports the baseline at a single fixed candidate order. What is the baseline accuracy averaged over at least 5 random orderings?