acceptodds
Withdrawn from ICLR 2027

Beyond Uniform Granularity: Benchmarking Mixed-Granularity Open-Set Recognition

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Most existing open-set recognition (OSR) benchmarks implicitly assume uniform category granularity across known class labels. In practice, however, known label sets often span multiple levels of category granularity due to differences in annotator expertise and other practical factors, resulting in a mismatch between such benchmarks and real-world recognition settings. To address this mismatch, we introduce Mixed-Granularity Open-Set Recognition (MG-OSR), where known classes may lie at different levels of an underlying category hierarchy while satisfying an antichain constraint, yet the hierarchy itself is not exposed to the model. We establish an MG-OSR benchmark and evaluate a broad range of conventional and vision-language-model-based OSR methods. We find that this mixed setting poses a threefold challenge to existing methods: (i) unseen subclasses of coarse-grained known classes are prone to being rejected as unknown; (ii) unknowns neighboring fine-grained known classes are prone to being accepted as known; and (iii) shared representation learning exhibits cross-granularity interference, reflected in reduced gradient coherence among sibling categories under the same semantic parent. We further show that widely used metrics, including AUROC and OSCR, are unreliable indicators of granularity-dependent recognition quality in mixed-granularity settings, motivating a granularity-aware evaluation metric that we use to systematically reassess existing methods. Our study identifies mixed-granularity label spaces as an overlooked regime of OSR and provides a unified testbed for OSR beyond uniform granularity.

Primary area:
datasets and benchmarks
Keywords:
Open-set recognition, mixed-granularity, benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_Jd6aOpenReview ↗
Rating
1
Confidence
3
Informativeness
3
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces Mixed-Granularity Open-Set Recognition (MG-OSR), a setting in which known labels may occupy different levels of an underlying category hierarchy while satisfying an antichain constraint, and the hierarchy itself is hidden from the learner. The paper argues that this regime changes the meaning of “unknown”: an unseen subclass of a coarse-grained known class should still be accepted, whereas a neighboring category outside a fine-grained known class should be rejected. The main contributions are the MG-OSR formulation, a controlled benchmark construction, an analysis of asymmetric failure modes and cross-granularity interference, and a granularity-aware evaluation protocol intended to expose failures that aggregate OSR metrics such as AUROC and OSCR can obscure. Validation is framed as a benchmark-based diagnostic and re-evaluation of existing conventional and VLM-based OSR methods rather than as a new recognition method.

Critical strengths

Strength 1: The conceptual reframing of “unknown” relative to the scope of known labels is clear and potentially impactful. The paper makes a simple but important point: being unseen during training does not uniquely determine whether a sample should be treated as unknown; the answer depends on the granularity at which the known label space is defined. This is consequential because it identifies a blind spot in standard OSR evaluation and gives the community a precise vocabulary for discussing scope-dependent knownness.

Critical weaknesses

Weakness 1: The MG-OSR setting and benchmark construction have limited ecological validity, and the central conclusions may be partly artifacts of the chosen taxonomy and split protocol. The definition requires known labels to form an antichain and keeps the hierarchy hidden from the learner, while benchmark construction uses the taxonomy to decide which descendants are unseen known variants and which neighbors are near unknowns. Real open-set deployments often involve known labels with ancestor–descendant relations, partially available semantics, or ambiguous scope boundaries; the paper does not establish how often the antichain regime occurs or how conclusions change when it is relaxed. This matters because the paper’s main contribution is the claim that mixed granularity is an overlooked and important regime. If the regime is narrowly constructed and highly dependent on hidden taxonomic choices, the broader impact is weakened.

Weakness 2: The paper is primarily a benchmark and analysis paper, but the evidence chain from the proposed diagnostics to the strong claims about representation learning and evaluation is not fully established. The paper explicitly states that it focuses on exposing and evaluating mixed-granularity failures rather than solving them, which limits the methodological contribution. Moreover, the cross-granularity interference analysis reports reduced gradient coherence among sibling categories, but does not establish a causal link between this coherence measure and downstream OSR performance, nor does it rule out alternative explanations such as label noise, class imbalance, or data distribution shifts induced by the split construction. The H-Score is also threshold-dependent and still an aggregate summary. For a paper whose main claims are about failure modes and evaluation mismatch, this makes the evidence more suggestive than decisive, which is a substantive rejection concern.

Weakness 3: The manuscript as provided has missing or corrupted figures.

Questions for authors

The provided manuscript has a corrupted figure region: Page 5 is largely repeated “1”s, Figure 1 is missing, and Figures 2 and 3 are unreadable with truncated captions. Can you confirm whether this is a submission or compilation error and provide a complete, high-resolution manuscript? Without readable Figures 2 and 3, the central claims about cross-granularity interference and evaluation mismatch cannot be assessed.

How common is the antichain constraint in realistic open-set deployments? If known labels can have ancestor–descendant relations, how should MG-OSR be extended, and would the main failure modes reported here persist?

The benchmark construction uses the hidden taxonomy to define unseen known variants and near unknowns. How sensitive are the conclusions to the choice of taxonomic branches, coarse/fine composition, and hierarchy depth? Can you report results under alternative split constructions?

H-Score depends on the TPR95 operating point and the harmonic mean of coarse-variant acceptance and near-unknown rejection. How stable are the method rankings under other thresholds or alternative aggregation choices?

The cross-granularity interference analysis reports reduced gradient coherence among sibling categories. Is this coherence measure correlated with per-branch OSR performance, and can an intervention that changes coherence produce a corresponding change in recognition behavior?

Given that no recognition method is proposed, how do the authors position the methodological contribution of this paper relative to benchmark and analysis papers? This directly affects how the contribution should be weighed.

Reviewer_z9sNOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a benchmark for open-set recognition, with particular focus on mixed granularity. The authors propose a new setting where known classes have different levels of granularity, which may cause difficulty for OSR. The paper constructs benchmarks for this setting using existing datasets and evaluates existing methods, and identifies three challenges. They also propose an evaluation metric to reflect these challenges.

Critical strengths

The paper studies a practical problem where known classes may be at different levels of category hierarchy, which may pose challenges for open-set orcognition.

Critical weaknesses
  1. This submission appears to be incomplete. All figures are missing. However, this is a benchmark paper for OSR, which is an image-based vision task, and also a large part of the claims build on visual evidence. It is very hard to evaluate this paper and validate its claims given the missing content.

  2. Also the evaluation metric does not fully reflect the performance of the intended task. In particular, while it is stated in the task definition (Eq. 4) that the model should predict the object class for known classes, the proposed H-Score does not capture this.

Questions for authors

See weaknesses above. The main reason is that all figures are missing in the main paper, making it clearly not ready for acceptance.

Reviewer_JjwwOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes new benchmarks for open-set recognition with mixed-granularity, where the classes may lie at different levels of an underlying category hierarchy. The authors also propose granularity-aware evaluation metrics to access the OSR methods.

Critical strengths

The idea of mixed-granularity benchmarks is quite interesting. The mixed-granularity benchmarks reveal new challenges in OSR, such as unseen subclasses of coarse-grained known classes are prone to false rejection and unknowns neighboring fine-grained known classes are prone to false positive.

Critical weaknesses

Apart from the concerns below the draft’s main issue is the missing figures (Figures 1, 2, and 3), which makes it difficult to evaluate the results and makes the submission incomplete. I tried with different browsers but failed to visualize the figures.

Questions for authors
  1. “This assumption can be mismatched with real-world labeling practice” – The authors should elaborate on the motivation behind introducing such benchmarks, for example in which use-cases such mixed-granularity may arrive.
  2. How to make a benchmark anti-chained if it is not anti-chained in the benchmarks raw/original form? For example, due to some labelling errors or labeling concepts at different levels of specificity by the annotators (or arguing with the same motivation posed by the authors) some known categories may subsume other categories in the dataset's original form.
  3. The authors should discuss in terms of how these benchmarks can reveal additional failure modes given a certain OSR method compared to fully fine-grained benchmarks, such as California birds, Aircraft or the benchmarks provided by Lang et al 2024.
  4. Before moving on to the metrics for granularity aware unknown rejection, the authors should also discuss how to evaluate the closed set problem of such a “specific” mixed-granularity supervised problem. The OSR problem is not only about the unknown class-rejection but also about the closed-set problem and the overall closed-set accuracy is not sufficient for such mix-granularity classification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.