acceptodds
Under review as a conference paper at ICLR 2027

OpenSight: Explainable Open-Set Fine-Grained Recognition with MLLMs

Abstract

Multimodal large language models (MLLMs) have demonstrated strong general capabilities. Still, they remain unreliable in open-set fine-grained visual recognition, where the correct decision is sometimes to reject all provided candidate categories. These tasks often rely on subtle visual distinctions to tell different categories apart. Through controlled analysis, we first find that perception is not the dominant reason for failed recognition. After extracting visual features, models often make decisions based on memorized category priors rather than grounded visual cues, leading to unfaithful reasoning. To this end, we introduce OpenSight, a reasoning-guided reinforcement-learning pipeline that curates samples with verified, attribute-grounded reasoning rationales to enhance fine-grained reasoning in MLLMs. By explicitly identifying hard negatives via subtle visual contradictions, OpenSight enables the model to ground its textual explanations in specific visual evidence. Importantly, the acquired reasoning pattern can be generalized to domains that are visually distinct from the training domain. Across six fine-grained open-set benchmarks, our approach improves the average accuracy by 4.1% and 13.0%, and the harmonic mean by 16.8% and 13.6% over standard GRPO-tuned and base Qwen2.5-VL, respectively. Qualitatively, we show that OpenSight facilitates more evidence-linked explanations and more reliable abstentions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Reasoning, MLLM, Explainability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_d9aFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the challenge of open-set visual recognition, specifically in scenarios where the correct category is absent from the available options. Using bird species recognition as the primary testbed, the authors perform controlled diagnostic tests. By querying the model on bird body part colors, evaluating object-level localization, and visualizing attention for color-related tokens, they find that raw visual perception is not the bottleneck. Instead, multimodal large language models (MLLMs) fail because they rely heavily on memorized textual category priors rather than grounding their answers in observable visual traits, leading to hallucinated matches.

To resolve this, the paper introduces OpenSight, a reasoning-guided reinforcement learning (RL) pipeline. The methodology relies on a GPT-5-driven data curation process broken into three steps: attribute extraction, visual attribute verification, and reasoning generation. To teach the model to reliably reject incorrect options, the pipeline incorporates contrastive explanations detailing why an image does not match hard-negative candidate species, which are mined using CLIP nearest-neighbor embeddings. A subset of 200 curated data samples was validated by human annotators to ensure quality.

Finally, the authors use Group Relative Policy Optimization (GRPO) to train the Qwen2.5-VL model. The RL reward function consists of three components: a format reward, an answer reward, and a reasoning reward. Across six fine-grained open-set benchmarks, the OpenSight approach improves average accuracy by 4.1% and 13.0%, and the harmonic mean by 16.8% and 13.6%, over standard GRPO-tuned and base Qwen2.5-VL models, respectively.

Critical strengths
  1. The paper is interesting and well motivated.
  2. The authors methodically isolate the root cause by demonstrating that the failure stems from reliance on memorized language priors during decision-making, rather than a fundamental lack of visual perception.
  3. The paper focuses on interpretability, verifying the model's attention at various stages.
  4. The results are encouraging, OpenSight approach improves the model performance on all datasets.
  5. Incorporating hard-negative explanations is an effective strategy.
Critical weaknesses
  1. The approach is highly structured and relies heavily on the availability of predefined taxonomic descriptions like different parts of the bird and coloration. These may not be available in all reasoning domains.
  2. The dataset generation relies heavily on MLLMs for reasoning which is then used for computing the dense reasoning reward. Since the ground truth reasoning is not a 100% free of hallucinations, the reward and the RL loop are susceptible to co-variate shift in training.
  3. Although the appendix provides some qualitative examples, the paper is lacking a systematic failure analysis section.
  4. The experiments are restricted to the Qwen2.5-VL model family. It remains unclear whether the reasoning-guided GRPO pipeline will generalize smoothly to other major MLLM architectures (eg LLaVA, Gemma etc).
Questions for authors
  1. Section 4.1 notes that human evaluators preferred the proposed pipeline's explanations by "28% over direct prompting". Could you report the full breakdown and an inter-annotator agreement metric?
  2. Could the authors provide a systematic, quantitative breakdown of OpenSight's failure modes on the test sets? It would be interesting to see the frequency of overusing the "None of the above" option.
  3. How were same species images removed during the nearest-neighbor matching process?
Reviewer_PhZbOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This work introduces a GRPO-based reinforcement learning strategy designed to boost the performance of models on fine-grained classification tasks.

Critical strengths
  1. The paper is clearly presented and well-written, accompanied by polished and aesthetically pleasing figures and tables.
Critical weaknesses
  1. Missing comparisons with frontier proprietary models & saturated benchmarks: The evaluation lacks comparisons against state-of-the-art closed-source foundation models. The selected benchmarks appear to have largely been saturated or solved by modern proprietary models; evaluating on more challenging, contemporary benchmarks such as WorldVQA would be far more convincing.
  2. Absence of contemporary fine-grained baselines: The paper does not benchmark against any recent domain-specific fine-grained classification methods; the comparisons are narrowly restricted to GRPO-based training variants.
  3. Limited methodological novelty: The proposed contribution appears to be largely an engineering heuristic focused on reward shaping within the standard GRPO framework, offering limited algorithmic novelty.
Questions for authors
  1. Why were comparisons against existing, recent fine-grained visual classification (FGVC) methods completely omitted?
  2. Aside from task-specific reward engineering, are there any other algorithmic or theoretical innovations introduced relative to standard GRPO?
  3. I strongly suggest including evaluation results on the WorldVQA benchmark and comparing against competitive methods currently featured on its leaderboard.
Reviewer_fxeqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Brief Summary: The paper tackles the task of fine-grained image classification using VLMs (qwen2.5-VL). For an input image and candidate categories, it grounds parts for each candidate and then gives the answer with an option for none-of-the-above. The main idea is that just by fine-tuning on bird-categories, the model can generalize to other non-bird datasets. For training, authors get gpt5 reasoning outputs, and use qwen3-4b as a judge to compare the gpt-5 reasoning output with that to qwen2.5-VL and then train it via GRPO. Experiments on multiple datasets such as cases with P=20,30 show the proposed method outperforms direct GRPO on qwen2.5-VL often by +10 pts on harmonic mean.

Critical strengths

Pros:

  1. Fine-grained classification with VLMs is still under-explored in the literature, and as such the paper is well motivated.

  2. The overall idea of trying to distinguish between close negatives with fine-grained classes makes sense. The authors provide detailed experimental results and ablation studies. In particular, the finding that training on bird datasets improves on other non-bird datasets is quite interesting.

  3. Authors conduct human-evals, have detailed ablation studies and additional experiments in supplementary along with visualizations.

Critical weaknesses

Cons:

  1. My main concern is that it is difficult to isolate where the gains are coming from. It seems to be that the paper is essentially distilling GPT-5 indirectly using Qwen3 as the judge to compare and give rewards, and as such doesn't seem like a scalable strategy as it relies on an external model to provide ground-truth reasoning? Please correct me if I am wrong here.

  2. It is unclear why authors are using Qwen2.5-VL? Why not just use Qwen3-VL? Whats the main reason to Qwen2.5-VL? Also, why use GPT-5 instead of more recent models like GPT-5.5?

  3. The experimental setup is skewed to favor the authors approach. For instance, CUB has ~200 classes but in their setup, it is assumed there is a first stage of down-filtering to give you candidate classes already and their model does finer grained classification. This is fine but this needs to be qualified with a two stage approach. See for instance [Ref1]

  4. It is unclear why authors don't go for a tool-calling option which can do zoom/crop with python sandbox for the images. Direct GRPO on Qwen with tool-calling should be explored and provided as a baselines.

  5. One of the selling points of the paper is supposed to be transfer from bird to non-bird datasets. But it seems like basic Qwen GRPO also transfers? This is more like basic task transfer than anything to do with fine-grained grounding?

===========

[Ref1]: Kumar, Raja, Arka Sadhu, and Ram Nevatia. "DiVE-k: Differential Visual Reasoning for Fine-grained Image Recognition." In International Conference on Learning Representations, vol. 2026, pp. 106415-106433. 2026.

Questions for authors

see weakness section above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.