acceptodds
Withdrawn from ICLR 2027

Seeing but Struggling to Say: Organizing Dispersed Evidence in Multi-modal Large Language Models for Fine-Grained Recognition

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

14 views

Abstract

Multi-modal large language models (MLLMs) have shown remarkable general capabilities, yet still struggle with fine-grained visual recognition (FGVR), a long-standing task that demands distinguishing subordinate-level categories through subtle visual cues. While existing efforts focus on improving MLLMs for FGVR by introducing external supervision, we revisit this challenge by asking whether MLLMs already possess fine-grained understanding intrinsically. Through layer-wise linear probing, we find that the fine-grained evidence is indeed present across the hidden states, but can hardly be decoded by the native generation. Furthermore, the evidence disperses away from the answer position at intermediate layers, and this dispersion hurts fine-grained recognition far more than coarse-grained one. What MLLMs lack is thus not the fine-grained knowledge, but its organization into a discriminative structure that the readout can exploit. Motivated by this, we propose **O**rganizing **A**ggregated fine-grained evidence with **S**elf-el**I**cited attribute**S** (OASIS), a simple yet effective method that aggregates the dispersed evidence at the middle layer and organizes it by attributes self-elicited from the MLLM itself. With the backbone entirely frozen, OASIS unlocks the intrinsic fine-grained capability of MLLMs across six benchmarks and four backbones, while mitigating the intermediate-layer dispersion and instilling fine-grained discriminative structure. The code is available at https://anonymous.4open.science/r/OASIS_MLLM.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Fine-grained Understanding, Vision-langugae Model, Fine-grained Attribute-weighted

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_W7GEOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies why MLLMs perform poorly on fine-grained visual recognition. The paper provides a layer-wise linear probing experiment and observes that fine-grained category information is decodable from visual tokens but becomes less accessible at the answer position in intermediate layers. Based on this observation, they propose OASIS, which uses a learnable summary token and a lightweight Transformer adapter to aggregate hidden-state information. Experiments on six benchmarks show consistent improvements over supervised-learning baselines.

Critical strengths
  1. The paper is well-organized and easy to follow.
  2. The reported experimental results are promising, although several important issues require further clarification and validation.
Critical weaknesses
  1. The motivation is not novel. Finedefics (ICLR 2025) has already investigated the discrepancy between MLLMs' ability to understand fine-grained visual attributes and their ability to identify the correct category. OASIS also shares important similarities with Finedefics (ICLR 2025), as both use supervised recognition training together with attribute-related contrastive learning to improve fine-grained visual recognition. Why is Finedefics not explicitly compared in the experiments?
  2. OASIS introduces an additional trainable module into the MLLM's forward pass to improve fine-grained recognition. However, the paper does not examine whether this module affects the model's original capabilities. The paper should clarify the practical advantages of the proposed approach. Conventional closed-set visual recognition models with 0.1B parameters can already achieve strong fine-grained classification performance. If OASIS improves fine-grained recognition at the cost of the MLLM's general multimodal capabilities, it essentially turns the MLLM into a task-specific classifier. In that case, the practical value of using an MLLM rather than a conventional discriminative model becomes questionable.
Questions for authors

Overall, I have many problems with the experiments:

  1. Why do the authors not report the linear probing performance of features extracted directly from the vision encoder, before they are fed into the LLM? In my experience, linear probing on frozen visual features can already achieve strong performance when the full training set is available. Without this comparison, it is unclear how much fine-grained discriminative information is actually contributed by the LLM
  2. The last token of the input prompt is used to conduct the probing experiment. Why aren't the first generated token or a fixed answer prefix used for comparison? How sensitive are the probing results to this choice?
  3. Figure 1 shows that the answer-position probe accuracy decreases at intermediate layers but recovers in the final layers. How does this observation lead to the claim that "such late and passive re-aggregation is suboptimal, as the recovered representation is neither complete nor in a form the native readout can exploit"? The probing results only measure linear separability at different layers.
  4. The claim of "without external priors" is not convincing. The attribute extraction prompts rely on manually predefined, dataset-specific attribute sets. These are clearly domain-specific priors. The use of predefined attribute sets also limits the practicality of the method.
  5. Table 1 reports 0% zero-shot accuracy for all four backbones on Stanford Cars. The authors attribute this to the strict make-model-year output format. I assume the paper uses open-ended generation for evaluation. However, given that the experiments are conducted on closed-set classification benchmarks, why not provide candidate class names in the prompt? Previous works such as Finedefice (ICLR 2025), Fine-R1 (ICLR 2026), and DiVE-k (ICLR 2026) have reported meaningful recognition performance on Stanford Cars. Moreover, in my experience, the strict output format requirement can be partially addressed by explicitly specifying the expected format in the prompt.
  6. The reported SFT performance, particularly on the Qwen models, seems lower than expected given the use of the full training sets. The authors should provide more details about the SFT implementation. It is important to establish that the improvement of OASIS does not mainly come from an insufficiently optimized SFT baseline. If OASIS is trained for N steps in the main experiments, please show the 2N-step or 4N-step performance curves throughout training.
Reviewer_oywPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The author conducts experiments and finds that for fine-grained visual recognition tasks, the VLM does not lack the ability to recognize, but the feature aggregation mechanism makes the outputs tokens fail to do fine-grained recognition. In order to solve this problem, the author proposes OASIS, by injecting summary tokens and do attribute guided contrastive learning to keep the fine-grained feature for recognition. And experiments demonstrates the effectiveness of this method.

Critical strengths
  1. The author do empirical study to explore the lack of fine-grained recognition ability in different layers by using linear probe, and the experiments reveals that an explict gap between the recoverable from hidden features. It is insightful for the mechanism why VLM performs bad at fine-grained recongnition tasks.
  2. The proposed OASIS method is parameter-efficient, and can be quickly adapted to current VLMs.
  3. The paper is well written and easy to understand.
Critical weaknesses
  1. The empirical study of the evidence of lacks information for fine-grained recognition may not well support the generation-based ability of the VLM for visual recognition since the mechanism, i.e., classification vs native generation, differs. And OASIS introduces classification and contrastive learning training, thus the performance gains may not come from the recovery of the information.
  2. Although the paper reveals the insightful new evidence of the loss of information, the methodogy itself still reuses current commonly used method like adapter, summary token, and attribute guided learning. Thus the novelty of the method is limited.
  3. Although the author compares OASIS with lora, full parameter finetuning, and prompt tuning. The proposed OASIS uses contrastive learning and additional adaptar, whereas the other method typically uses classfication objective, and the comparison may not fair.
Questions for authors
  1. Can the author provides more evidance of whether the performance gains of the method comes from the recovery of needed information, or introduces additional discriminative representation?
  2. Can the author provides more fair comparision with current methods.
Reviewer_qaHfOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why MLLMs struggle with fine-grained recognition despite encoding discriminative visual information. The authors propose OASIS, which uses a learnable summary token to aggregate dispersed evidence and self-generated attributes to guide contrastive learning, while keeping the backbone frozen. Experiments across six benchmarks and four MLLMs demonstrate consistent improvements over fine-tuning baselines.

Critical strengths

1. Interesting motivation. The paper identifies an interesting gap between the fine-grained information encoded in MLLMs and their actual recognition performance. Through layer-wise probing, the authors provide empirical evidence for this observation and propose OASIS to address the issue.

2. Solid experiments. The experiments cover various benchmarks and four MLLM backbones, with consistent improvements over full fine-tuning and parameter-efficient baselines.

Critical weaknesses

1. Limited generalization. The attribute-guided training relies on labels from the training categories, and evaluation is limited to the same categories. Since the LLM is frozen, OASIS seems more like a task-specific adapter than a general improvement to MLLM capabilities. Its generalization to unseen categories remains unclear.

2. Unclear role of the summary token. The summary token is used for contrastive learning but is not directly passed to the subsequent LLM layers. Instead, its information is indirectly transferred to the original hidden states through bidirectional attention. I am not fully convinced that this mechanism effectively injects the aggregated discriminative information. More direct evidence would strengthen this claim.

Questions for authors
  1. Additional ablations. Could the authors investigate: (1)Unfreezing the LLM backbone, as joint fine-tuning might further improve performance. (2)Different numbers of summary tokens. (3)Aggregation at earlier or later layers, or across multiple layers.
  2. Generalization. Could the authors evaluate OASIS on unseen categories or cross-dataset transfer to demonstrate that the learned representations generalize beyond the training categories?
  3. The authors find that the generative classification performance of MLLMs is significantly worse than that of linear probes, but the underlying reasons are not fully explored. Could there be other explanations beyond evidence dispersion? For example, does visual information gradually shift to other text token positions in deeper layers, or is there a mismatch between discriminative representations and the generative objective? It would be helpful if the authors could further discuss or investigate these possibilities.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.