acceptodds
Withdrawn from ICLR 2027

Generative Vision-Language Models Can Discriminate: Unlocking Zero-Shot Classification Potential

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Contrastive Vision-Language Models (VLMs), such as CLIP, are naturally suited to zero-shot classification through image–text similarity matching, whereas generative VLMs provide an autoregressive decoding interface that is less direct for closed-set classification. Existing protocols based on open-ended generation or candidate-constrained decoding can be sensitive to label wording, output format, and prompt design. This motivates us to examine whether part of the observed performance gap arises from the inference interface rather than representational capacity alone. In this paper, we propose a zero-shot framework that unlocks the discriminative potential of generative VLMs. By inducing a contrastive-compatible latent space directly from the model's internal representations, we establish a similarity-based matching mechanism that mimics the contrastive formulation, while preserving the rich semantics of the generative paradigm. Furthermore, we identify a phenomenon where the VLM's internal linguistic priors accurately recognize visual concepts but diverge from canonical label taxonomies. To mitigate this, we introduce semantic equivalence rectification, which aligns diverse linguistic manifestations to ground-truth categories. Experiments across multiple benchmarks show consistent improvements over generative prompting baselines. Further analyses identify taxonomy–class alignment as an important, but not exclusive, source of SER's gains.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language Models, Zero-Shot Image Classification, Multimodal Foundation Models, Representation Learning, Closed-Set Recognition

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_3D9NOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks a simple but important question: are generative vision-language models (VLMs) bad at zero-shot image classification because their visual understanding is weak, or because asking them to generate a label one token at a time is the wrong way to get a classification answer out of them? The authors argue that much of the gap lies in this inference interface, not in the model's representations. Their solution is to sidestep generation altogether. Instead of prompting the model to output a class name, they extract its internal hidden states at the moment it is "about to" name the object — once from an image prompt, and once for each candidate class described in text — and then classify by simple similarity matching. A second component, Semantic Equivalence Rectification (SER), enriches each class with LLM-generated aliases, scientific names, and descriptions, and only activates this expanded vocabulary when the model is uncertain. The approach is tested on eight benchmarks and seven generative VLMs. Latent matching consistently outperforms a reproduced closed-world prompting baseline.

Critical strengths
  • The distinction between representational capacity and inference-interface limitations is important. As generative VLMs are increasingly deployed as general-purpose multimodal systems, enabling reliable fixed-output classification without fine-tuning or an additional contrastive model has clear practical value.
  • The proposed latent-matching procedure is architecture-agnostic and requires no parameter updates. It could plausibly be applied to a range of instruction-tuned VLMs without retraining or benchmark-specific adaptation.
  • The paper evaluates seven generative VLMs from multiple model families on eight benchmarks, including fine-grained recognition, general object classification, food, textures, etc. The consistency of the improvement across most settings strengthens the empirical benefits.
  • The investigation of Taxonomic Bias is one of the paper’s more original contributions. The wrong-class mapping experiment is a meaningful attempt to determine whether SER is genuinely exploiting taxonomy–class correspondence rather than simply adding more text.
Critical weaknesses
  • The reproduced closed-world baseline requires the model to generate one label and then parses the response. This does not exhaust the reasonable ways to perform classification with a generative VLM. Stronger and more direct baselines should include, at minimum:

    • ranking candidates by their token or sequence probabilities rather than relying on a generated answer (the "Prob Inference" variants in Zhang et al. 2024);
    • constrained decoding restricted to candidate labels;
    • multiple prompt templates and answer-format instructions;
    • comparison with recent probability-based or calibrated classification protocols for generative VLMs, e.g., [1][2][3]

    The extremely low baseline scores for some model–dataset pairs—for example, approximately 0% or 0.05% on Cars196—suggest that prompt construction, candidate enumeration, response formatting, or parsing may be responsible for a substantial part of the observed gap.

  • SER uses an external LLM to generate class-specific scientific names, aliases, synonyms, and descriptions. Although no target images or image–label pairs are used, this is still an inference-time semantic prior derived from external knowledge. It is not directly comparable to a method using only benchmark class names, [4] and [5] are comparable semantic augmentation works. The paper correctly reports “Ours” and “+SER” separately, but the abstract and some conclusions still present SER as part of the general framework without consistently emphasizing this additional knowledge source. Moreover, the baseline should also be evaluated with the same synonym-aware answer mapping; otherwise, SER receives an advantage unavailable to the generative baseline.

  • Max pooling over externally generated variants can favor classes with more expressive, generic, or numerically extensive descriptions. The audit itself shows that unrelated broad expansions recover 59 of 113 corrected samples, which weakens the interpretation of SER as purely taxonomy-aware rectification. The wrong-class mapping experiment is useful, but it does not fully separate taxonomy alignment from prototype geometry and generic description effects.

  • The geometric argument establishes only that same-token contexts face related constraints in the space visible to the LM head; it does not imply cosine proximity. The controlled probe is text-only and therefore does not directly validate cross-modal visual–textual alignment. The main classification results show task-level utility, but they do not isolate whether matching works because of shared semantics, shared prompt syntax, modality-specific artifacts, or spurious correlations.

References

[1] Multimodal Large Language Models as Image Classifiers, ICLR 2026.
[2] On Large Multimodal Models as Open-World Image Classifiers, ICCV 2025.
[3] Revisiting MLLMs: An In-Depth Analysis of Image Classification Abilities, ArXiv 2024.
[4] Visual Classification via Description from Large Language Models, ICLR 2023.
[5] What does a platypus look like? Generating customized prompts for zero-shot image classification, ICCV 2023.

Questions for authors
  1. Did the authors evaluate the strongest inference protocol available on their own reproduced baseline: scoring each candidate label by its conditional token probability and taking the argmax (the "Prob Inference" variants evaluated in Zhang et al. 2024)?
  2. What happens when the same Gemini-generated aliases, scientific names, and descriptions are made available to the closed-world baseline — e.g., as answer-normalization vocabulary, or as additional enumerated options?
  3. The reproduced baseline scores 0.00% (LLaVA-NeXT, Cars196) and 0.05% (LLaVA-v1.5, Cars196). For these near-zero cases, what fraction of outputs were successfully parsed, and did the authors manually inspect a sample of failures?
  4. How many tokens does the 1,000-class closed-world prompt occupy for each model, and did every backbone process it without context truncation or degraded attention?
  5. Is the number of variants M identical for every class within a dataset, and are the variants' lengths roughly comparable?
  6. The paper invokes permutation-order sensitivity of enumerated lists as motivation, yet reports the baseline at a single fixed candidate order. What is the baseline accuracy averaged over at least 5 random orderings?
Reviewer_CuvaOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the sensitivity of generative VLMs to text formats in zero-shot classification by bypassing autoregressive generation and directly extracting last-layer hidden states as visual and textual embeddings for cosine similarity matching. To fix mismatches between model vocabulary and target labels, it introduces Semantic Equivalence Rectification (SER) to dynamically aggregate similarity scores across LLM-generated synonyms and scientific names. Across multiple benchmarks and VLM backbones, this approach significantly outperforms standard closed-world prompting baselines without any model fine-tuning.

Critical strengths
  1. This method is computationally efficient and training-free, operating on fully frozen VLM backbones and pre-computing cached textual embeddings offline to avoid processing long candidate lists in prompts, thereby reducing inference context length and overhead.
  2. The paper presents a comprehensive empirical evaluation across seven different generative VLM backbones and eight diverse image classification datasets. This broad experimental setup convincingly demonstrates that the proposed method is architecture-agnostic and generalizes well across various domain taxonomies.
Critical weaknesses
  1. The practical necessity of using generative VLMs for closed-set zero-shot classification is not strongly justified. As shown in Tables 10 and 11, the contrastive models like CLIP consistently and significantly outperform the proposed method across major benchmarks. Given that contrastive VLMs utilize a much simpler dual-encoder architecture that offers higher throughput and lower computational overhead, it remains unclear why a practitioner would deploy a significantly heavier generative backbone solely for image classification tasks.
  2. The method assumes that hidden states associated with predicting the same next-token are sufficiently close in representation space to be compared via cosine similarity. However, sharing a target prediction token does not mathematically or geometrically guarantee vector proximity in the hidden space. Furthermore, unlike contrastive models, generative VLMs are trained via cross-entropy rather than a cosine-similarity objective, making it unclear why cosine distance provides a reliable classification metric. Additionally, extracting only the last-layer hidden state immediately preceding the first answer token raises concerns that the observed similarity captures shallow, token-level prediction patterns (e.g., shared initial tokens across distinct classes) rather than true image-text visual semantics.
  3. The proposed Semantic Equivalence Rectification (SER) mechanism closely resembles existing prompt enrichment strategies. For instance, prior works such as CuPL have already demonstrated the utility of using LLM-generated class descriptions to improve zero-shot visual classification. The paper does not adequately differentiate SER from these established approaches or explicitly delineate its technical novelty beyond standard text-prototype expansion.
  4. The main experimental comparisons rely heavily on a closed-world autoregressive prompting baseline that yields near-zero accuracy in several settings (e.g., Stanford Cars). This creates an artificially weak reference point and makes it difficult to discern whether the reported gains stem from the proposed hidden-state matching or simply from avoiding a dysfunctional generation protocol. To properly isolate the contributions of this work, the paper should compare against stronger logit-based baselines—such as candidate log-likelihood scoring or calibrated output-head scoring—which constrain the prediction space without requiring hidden-state cosine matching.
  5. Parenthetical citations (\citep) and narrative citations (\citet) are frequently mixed up (e.g., in Section 1 and Section 2), causing widespread grammatical and formatting errors. Please carefully review and fix all LaTeX citation tags.
Questions for authors
  1. What is the practical advantage of using a generative VLM for classification instead of a contrastive model such as CLIP?
  2. Why should hidden states associated with the same next-token prediction be close under cosine similarity? Have the authors investigated cases where different categories share the same first token, or where semantically equivalent expressions have different first tokens? More direct cross-modal representation analyses would be helpful.
  3. How does the proposed method compare with direct candidate log-likelihood scoring or calibrated logit-based classification using the same generative VLM? Why is cosine similarity between hidden states preferable to using the model's output probabilities?
  4. How is the confidence threshold in the adaptive cascade selected for each model and dataset? Does its selection require labeled validation data, and how is this consistent with the claimed zero-shot setting?
Reviewer_YfL7OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies zero-shot closed-set classification with frozen generative VLMs. The authors take hidden states from image prompts and text prompts before the model generates an answer. They then classify images by cosine similarity. A module called Semantic Equivalence Rectification (SER) adds externally generated class names and descriptions to the class anchors. An adaptive cascade turns on this expansion only when the prediction margin is low. The experiments use seven VLMs and eight datasets.

Critical strengths
  1. The method reuses a frozen generative VLM for classification and trains no new parameters. This is useful in practice.
  2. The evaluation covers several model families and datasets. In several settings the method clearly beats the reproduced direct-generation baseline.
Critical weaknesses
  1. Novelty is limited and not well separated from prior work. [1] already uses an LLM to write class descriptions. It then classifies images by comparing them with these descriptions. This paper mainly replaces CLIP embeddings with generative VLM hidden states. It also adds semantic expansion and margin-based routing. [2] already uses prompts to extract multimodal representations at the next-token position. The paper discusses neither work. It does not show a clear contribution beyond this extension.
  2. The ablations do not isolate the role of generative VLM features. The authors ablate the semantic bank and the routing. They never apply the same bank, aggregation rule and routing to CLIP. Table 10 only compares against a standard CLIP classifier, so it does not settle the question. Without matched tests on CLIP and stronger CLIP variants, the source of the gains is unclear. They may need generative VLM hidden states. They may also come from a general description-based strategy.
  3. The zero-shot setting ignores possible test-set exposure. Frozen weights and no labeled downstream training do not prove that the test images were absent from pretraining or instruction tuning. This concern applies to both VLMs and CLIP. It still limits how the results can be read, especially across backbones with different training data. The paper offers no contamination check. It also offers no test on newly collected images. Gains on the same backbone are still informative. They do not prove generalization to truly unseen images.
  4. Key implementation and calibration details are missing. The source of the main “[Description of class]” anchors is unclear, and so is how they are built. Appendix G states that Gemini 3 Pro generates the semantic bank. It does not give the full prompts or the banks. The paper also does not report the thresholds used in the main experiments or how they were chosen. Applying semantic expansion to every sample often hurts accuracy. Threshold selection therefore matters for both reproducibility and the zero-shot claim.
  5. The evidence for the representation mechanism is weak. The controlled probe uses only 32 text-only prompts in four categories. It does not test whether image and text hidden states are directly comparable. It also does not show that matched next-token prediction contexts are needed. The paper lacks comparisons with ordinary prompting, other extraction positions, other layers and other pooling methods. The paper shows that the chosen representations can support classification. It does not explain why this extraction design is necessary.

[1] Visual Classification via Description from Large Language Models.

[2] E5-V: Universal Embeddings with Multimodal Large Language Models.

[3] Why are Visually-Grounded Language Models Bad at Image Classification?

Questions for authors
  1. What is the main methodological contribution beyond applying description-based classification [1] to generative VLM features? How does it differ from the extraction approach in [2]?
  2. How do CLIP and stronger CLIP variants perform with the same semantic bank, aggregation rule, routing and threshold selection?
  3. Do the gains hold against class-name likelihood scoring and calibrated generative classification? Why do some baselines score near zero?
  4. Which evaluation datasets might overlap with the training data of each backbone? Can the authors provide an overlap check or results on newly collected images?
  5. How are the main descriptions generated? How are the thresholds chosen without target labels? Please give the exact prompts and settings.
  6. Under the same conditions, how much does matched next-token prompting improve over ordinary hidden-state extraction?

[1] Visual Classification via Description from Large Language Models.

[2] E5-V: Universal Embeddings with Multimodal Large Language Models.

[3] Why are Visually-Grounded Language Models Bad at Image Classification?

Reviewer_AJvqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper motivates the challenges of zero shot classification in generative vision language models, which predict using next token prediction instead of defining a contrastive representation space. It argues that open world generation (predicting class directly from the model) suffers from vocabulary mismatch and closed world prompting approaches struggle in managing context while providing the full class taxonomy as input to the model.

The paper proposes creating a latent comparable space using proposed visual anchor and textual anchor induction. For each anchor, the paper extracts the relevant embedding by taking the hidden state of the token immediately before the class prediction. The argument is that these representations are taken under matched target token prediction contexts, and therefore should be aligned. The paper uses normalzied cosine similarity for classification here.

Another problem that the paper highlights is what it refers to as taxonomic bias. Basically the VLM may have preference towards other names (e.g. synonyms) of the concepts in the classification taxonomy. Therefore, the paper develops a cascaded strategy which computes predictions by aggregating over the semantic set of names for each concept.

Experiments are done over several datasets like CUB 200, Oxford Flowers, Stanford Cars, ImageNet etc. The paper uses a single baseline, and ablates its design with and without the semantic cascaded inference.

Critical strengths
  1. The paper evaluates its approach adn design choices comprehensively. Several VLMs are evaluated to show that this is not just a method thath works on a specific VLM. Understanding the impact of SER is also helpful.
  2. The proposed method is simple and intuitive. At the same time, it is scalable compared to closed world approaches as it can increase its context set while maintaining the similarity based classification strategy.
  3. The paper lists a discussion on its limitation in short context scenarios. It is helpful to understand how changing the model could impact the effect of SER.
Critical weaknesses
  1. The paper only considers one closed world generation baseline. There are several papers, which could provide a useful baseline for this method, and incorporating at least one more would significantly strengthen the paper. E.g. [1, 2, 3] could be adapted to provide strong baselines.
  2. The paper misses discussion of previous literature using alternative names / synonyms for improving classification. There is literature like [4] which should be discussed. While these papers may not be on generative VLMs, the overall concept of improving taxonomy bias using alternative names holds.
  3. (Nit) It is hard to understand how related work section 2.1 is related to the paper. It would be great if the paper can include some context on what the paper contributes w.r.t. this related work.

References

  1. https://arxiv.org/abs/2306.01879, ICML 2024
  2. https://arxiv.org/pdf/2406.14496, EMNLP 2024
  3. https://arxiv.org/abs/2405.11301, EMNLP 2024
  4. https://arxiv.org/abs/2401.12425, CVPR 2024
Questions for authors

It would be great to understand the reasoning behind some of the empirical evidence. For example, the paper finds that always applying the semantic bank leads to a drop in performance, which feels a bit counterintuitive. It would be helpful if authors can share their insight on this. Currently the paper hypothesizes that it might be unnecessary noise (line 430), but having more insight would be great.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.