- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses the challenge of open-set visual recognition, specifically in scenarios where the correct category is absent from the available options. Using bird species recognition as the primary testbed, the authors perform controlled diagnostic tests. By querying the model on bird body part colors, evaluating object-level localization, and visualizing attention for color-related tokens, they find that raw visual perception is not the bottleneck. Instead, multimodal large language models (MLLMs) fail because they rely heavily on memorized textual category priors rather than grounding their answers in observable visual traits, leading to hallucinated matches.
To resolve this, the paper introduces OpenSight, a reasoning-guided reinforcement learning (RL) pipeline. The methodology relies on a GPT-5-driven data curation process broken into three steps: attribute extraction, visual attribute verification, and reasoning generation. To teach the model to reliably reject incorrect options, the pipeline incorporates contrastive explanations detailing why an image does not match hard-negative candidate species, which are mined using CLIP nearest-neighbor embeddings. A subset of 200 curated data samples was validated by human annotators to ensure quality.
Finally, the authors use Group Relative Policy Optimization (GRPO) to train the Qwen2.5-VL model. The RL reward function consists of three components: a format reward, an answer reward, and a reasoning reward. Across six fine-grained open-set benchmarks, the OpenSight approach improves average accuracy by 4.1% and 13.0%, and the harmonic mean by 16.8% and 13.6%, over standard GRPO-tuned and base Qwen2.5-VL models, respectively.
Critical strengths
- The paper is interesting and well motivated.
- The authors methodically isolate the root cause by demonstrating that the failure stems from reliance on memorized language priors during decision-making, rather than a fundamental lack of visual perception.
- The paper focuses on interpretability, verifying the model's attention at various stages.
- The results are encouraging, OpenSight approach improves the model performance on all datasets.
- Incorporating hard-negative explanations is an effective strategy.
Critical weaknesses
- The approach is highly structured and relies heavily on the availability of predefined taxonomic descriptions like different parts of the bird and coloration. These may not be available in all reasoning domains.
- The dataset generation relies heavily on MLLMs for reasoning which is then used for computing the dense reasoning reward. Since the ground truth reasoning is not a 100% free of hallucinations, the reward and the RL loop are susceptible to co-variate shift in training.
- Although the appendix provides some qualitative examples, the paper is lacking a systematic failure analysis section.
- The experiments are restricted to the Qwen2.5-VL model family. It remains unclear whether the reasoning-guided GRPO pipeline will generalize smoothly to other major MLLM architectures (eg LLaVA, Gemma etc).
Questions for authors
- Section 4.1 notes that human evaluators preferred the proposed pipeline's explanations by "28% over direct prompting". Could you report the full breakdown and an inter-annotator agreement metric?
- Could the authors provide a systematic, quantitative breakdown of OpenSight's failure modes on the test sets? It would be interesting to see the frequency of overusing the "None of the above" option.
- How were same species images removed during the nearest-neighbor matching process?