acceptodds
Withdrawn from ICLR 2027

Beyond Naming: Re-benchmarking Open-World Classification with Discrimination

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Open-world classification seeks to recognize objects without a predefined candidate set. With MLLMs, this ability is realized and evaluated through free-form naming, based primarily on whether the generated name is correct and sufficiently specific. However, a model may name a category correctly while failing to distinguish it from visually similar classes. We introduce DOWC (Discrimination-Aware Open-World Classification), a benchmark that jointly evaluates naming and discrimination through textual and visual discrimination tasks. Our evaluation reveals that base models exhibit substantially weaker discrimination than naming, particularly in purely visual settings. To address this gap, we propose DBN (Discriminate Before You Name), which introduces an explicit local class comparison before naming. DBN first constructs a comparison neighborhood from self-proposed plausible classes, selects the candidate best supported by the image, and uses this selection to guide naming. We instantiate DBN in two forms: DBN-RL internalizes this process through supervised warm-up and reinforcement learning, with rule-based rewards coupling answer correctness and selection consistency, whereas DBN-TF applies it training-free to frozen models through coarse-to-fine candidate proposal and confidence-based selection. Across five base models, DBN-RL and DBN-TF improve discrimination by +11.0 and +9.4 points and naming by +7.8 and +7.1 points, respectively. These results show that explicit local discrimination provides a stronger basis for open-world recognition.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal large language models, open-world classification, visual discrimination, fine-grained recognition, benchmark, evaluation, vision-language models, naming

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xGTkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the gap between correct naming and reliable discrimination in open-world classification. The authors introduce DOWC, a benchmark evaluating both capabilities through textual and visual tasks, and propose the method DBN, which compares self-generated candidates before answering. Both training-based and training-free implementations improve discrimination and average naming scores across multiple multimodal models.

Critical strengths

The strengths are listed as follows:

  • The paper presents a meaningful limitation of naming-based evaluation and provides a structured benchmark for evaluating fine-grained discrimination. The combination of textual and visual tasks, together with the taxonomic evaluation, offers additional diagnostic information about model behavior.
  • Both training-based and training-free implementations are evaluated across multiple multimodal models, with fine-grained analyses of model behavior.
Critical weaknesses

However, I have the following concerns:

  • My primary concern is that the improvements may partly result from additional supervision or inference computation. DBN-RL receives naming and discrimination supervision, whereas SpeciaRL primarily targets naming. Meanwhile, DBN-TF introduces additional inference calls and thus leads approximately 5.8–6.7 times the latency of direct inference. The authors are encouraged to compare against multitask GRPO using the same training queries without discrimination notes, and inference baselines such as independent naming followed by answer matching under a comparable computational budget.

  • The conclusion that DBN improves naming requires further clarification because the naming metrics do not improve consistently. For example, on the fine-grained group with qwen2.5 7B, DBN-RL increases SS, while other metrics such as TI, LI, and CS all decrease. The improvement in the average score therefore does not necessarily indicate more accurate class identification. The authors are suggested to discuss these disagreements.

  • The benchmark construction depends on a single visual encoder for selecting confusable classes and matching positive and negative instances. This may make the measured difficulty sensitive to the encoder’s representation. The authors are sugested to evaluate alternative visual encoders and report results separately for endpoint cases and the remaining queries.

Questions for authors

please see the weakness above.

Reviewer_wo8fOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the problem of open-world classification with MLLM, and points out that, simply naming the object class correctly does not justify that the MLLM can correctly distinguish it from similar classes. To study this problem, the authors propose to construct a discrimination-aware benchmark, and propose a method named DBN to strengthen discrimination capability, with two implementations (DNB-RL and DBN-TF). Experiments on different MLLMs show the effectiveness of the proposed method.

Critical strengths
  1. The identified problem is practical, although it may not be very intuitive at first glance. In some cases MLLMs can correctly name the object without truly knowing how to distinguish it from other similar objects, and this can give false impression of the model's capability for open-world classification.

  2. The benchmark evaluates discrimination with four complementary tasks, providing a useful way to analyze discrimination across different levels of category granularity.

Critical weaknesses
  1. The performance improvement may come from training on the same question format. DBN-RL is trained on discrimination questions of the four task types, using the same question formats as the benchmark. But the base model and SpeciaRL are not trained on such questions. It is therefore not clear whether the performance gain come from training on the same formats or from the proposed design.

  2. The paper interprets low performance on the proposed discrimination tasks as evidence of limited visual discrimination. However, the results may not adequately isolate this effect, as the results can also be affected by question formulation and model response tendencies. For example, the base model tends to output long sentences while the proposed method tends to answer short class names. Such tendencies can also affect evaluation correctness and the justification of models' discrimination capability.

Questions for authors
  1. Table 3 shows that supervised warm-up already leads to a large part of the improvements on discrimination, while RL only brings relatively limited performance gain for some models. Further analysis would help clarify the contribution of the proposed RL component.

  2. The benchmark uses DINOv3 feature similarity to select confusable classes and visual instances. The benchmark difficulty may thus depend on this, and no other encoder is tested.

  3. There are also some inconsistencies. For example, Sec. 4.3 states increasing K “from 1 to 2” helps, but Fig. 5 starts at K = 2.

Reviewer_p99mOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this draft, the authors proposed DOWC, a benchmark that evaluates both naming and discrimination against visually similar classes for open-world classification with MLLMs. They also proposed DBN (Discriminate Before You Name), which proposes a few candidate names and selects one before answering, either trained with SFT + RL (DBN-RL) or training-free (DBN-TF). Experiments on five base models showed improvements of about 10 points on discrimination and 7 points on naming.

Critical strengths
  1. Evaluating naming and discrimination on the same images is interesting, and the benchmark design (paired verification, order swapping, hard negatives) is careful.
  2. DBN is cheap and practical, and the gains on the image-based tasks are large across all five base models.
Critical weaknesses
  1. It is not clear the gains come from the explicit candidate comparison. DBN-RL is trained on discrimination queries in the same format as the benchmark while the baselines are not, and DBN-TF seems to simply use the first generated name.
  2. The naming gains seem to come mostly from shorter outputs. For the stronger models, TI and LI drop while only SS goes up.
  3. The claim that correct naming does not imply discrimination is only shown by one example and averages, without a same-image analysis.
Questions for authors

Please refer to the Critical Weaknesses section for details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.