- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the gap between correct naming and reliable discrimination in open-world classification. The authors introduce DOWC, a benchmark evaluating both capabilities through textual and visual tasks, and propose the method DBN, which compares self-generated candidates before answering. Both training-based and training-free implementations improve discrimination and average naming scores across multiple multimodal models.
Critical strengths
The strengths are listed as follows:
- The paper presents a meaningful limitation of naming-based evaluation and provides a structured benchmark for evaluating fine-grained discrimination. The combination of textual and visual tasks, together with the taxonomic evaluation, offers additional diagnostic information about model behavior.
- Both training-based and training-free implementations are evaluated across multiple multimodal models, with fine-grained analyses of model behavior.
Critical weaknesses
However, I have the following concerns:
-
My primary concern is that the improvements may partly result from additional supervision or inference computation. DBN-RL receives naming and discrimination supervision, whereas SpeciaRL primarily targets naming. Meanwhile, DBN-TF introduces additional inference calls and thus leads approximately 5.8–6.7 times the latency of direct inference. The authors are encouraged to compare against multitask GRPO using the same training queries without discrimination notes, and inference baselines such as independent naming followed by answer matching under a comparable computational budget.
-
The conclusion that DBN improves naming requires further clarification because the naming metrics do not improve consistently. For example, on the fine-grained group with qwen2.5 7B, DBN-RL increases SS, while other metrics such as TI, LI, and CS all decrease. The improvement in the average score therefore does not necessarily indicate more accurate class identification. The authors are suggested to discuss these disagreements.
-
The benchmark construction depends on a single visual encoder for selecting confusable classes and matching positive and negative instances. This may make the measured difficulty sensitive to the encoder’s representation. The authors are sugested to evaluate alternative visual encoders and report results separately for endpoint cases and the remaining queries.
Questions for authors
please see the weakness above.