- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies why MLLMs perform poorly on fine-grained visual recognition. The paper provides a layer-wise linear probing experiment and observes that fine-grained category information is decodable from visual tokens but becomes less accessible at the answer position in intermediate layers. Based on this observation, they propose OASIS, which uses a learnable summary token and a lightweight Transformer adapter to aggregate hidden-state information. Experiments on six benchmarks show consistent improvements over supervised-learning baselines.
Critical strengths
- The paper is well-organized and easy to follow.
- The reported experimental results are promising, although several important issues require further clarification and validation.
Critical weaknesses
- The motivation is not novel. Finedefics (ICLR 2025) has already investigated the discrepancy between MLLMs' ability to understand fine-grained visual attributes and their ability to identify the correct category. OASIS also shares important similarities with Finedefics (ICLR 2025), as both use supervised recognition training together with attribute-related contrastive learning to improve fine-grained visual recognition. Why is Finedefics not explicitly compared in the experiments?
- OASIS introduces an additional trainable module into the MLLM's forward pass to improve fine-grained recognition. However, the paper does not examine whether this module affects the model's original capabilities. The paper should clarify the practical advantages of the proposed approach. Conventional closed-set visual recognition models with 0.1B parameters can already achieve strong fine-grained classification performance. If OASIS improves fine-grained recognition at the cost of the MLLM's general multimodal capabilities, it essentially turns the MLLM into a task-specific classifier. In that case, the practical value of using an MLLM rather than a conventional discriminative model becomes questionable.
Questions for authors
Overall, I have many problems with the experiments:
- Why do the authors not report the linear probing performance of features extracted directly from the vision encoder, before they are fed into the LLM? In my experience, linear probing on frozen visual features can already achieve strong performance when the full training set is available. Without this comparison, it is unclear how much fine-grained discriminative information is actually contributed by the LLM
- The last token of the input prompt is used to conduct the probing experiment. Why aren't the first generated token or a fixed answer prefix used for comparison? How sensitive are the probing results to this choice?
- Figure 1 shows that the answer-position probe accuracy decreases at intermediate layers but recovers in the final layers. How does this observation lead to the claim that "such late and passive re-aggregation is suboptimal, as the recovered representation is neither complete nor in a form the native readout can exploit"? The probing results only measure linear separability at different layers.
- The claim of "without external priors" is not convincing. The attribute extraction prompts rely on manually predefined, dataset-specific attribute sets. These are clearly domain-specific priors. The use of predefined attribute sets also limits the practicality of the method.
- Table 1 reports 0% zero-shot accuracy for all four backbones on Stanford Cars. The authors attribute this to the strict make-model-year output format. I assume the paper uses open-ended generation for evaluation. However, given that the experiments are conducted on closed-set classification benchmarks, why not provide candidate class names in the prompt? Previous works such as Finedefice (ICLR 2025), Fine-R1 (ICLR 2026), and DiVE-k (ICLR 2026) have reported meaningful recognition performance on Stanford Cars. Moreover, in my experience, the strict output format requirement can be partially addressed by explicitly specifying the expected format in the prompt.
- The reported SFT performance, particularly on the Qwen models, seems lower than expected given the use of the full training sets. The authors should provide more details about the SFT implementation. It is important to establish that the improvement of OASIS does not mainly come from an insufficiently optimized SFT baseline. If OASIS is trained for N steps in the main experiments, please show the 2N-step or 4N-step performance curves throughout training.