acceptodds
Withdrawn from ICLR 2027

Granularity-Aware Evidence Aggregation and Calibration for Training-Free Zero-Shot Multi-Label Recognition

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

CLIP provides strong zero-shot recognition capabilities, but its global representation may overlook small or spatially localized objects in multi-label images. Local crops can recover such evidence, yet may also amplify responses to absent classes, including semantically related alternatives. We introduce GRADE (Granularity-Aware Regional Aggregation and Discriminative Evidence Calibration), a training-free framework for zero-shot multi-label image recognition. GRADE aggregates class-specific regional responses across multiple context granularities, using class-wise global-score percentiles to guide aggregation. When local evidence dominates, comparison with semantically neighboring classes downweights ambiguous local responses before global-local fusion. Experiments on COCO and VG-256 demonstrate competitive performance against existing training-free methods, with consistent mAP improvements over global CLIP across both benchmarks. Ablation studies further confirm the effectiveness of aggregating local evidence across multiple regions and context granularities.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-Label Recognition, Vision-Language Models, Multi-Granularity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zCXHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes GRADE, a training-free framework for zero-shot multi-label image recognition. The authors observe that global CLIP representations may overlook small or localized objects, while local crops can introduce misleading class responses.

To address these issues, GRADE combines class-specific multi-region extraction, multi-granularity evidence aggregation, distribution-guided calibration, and semantic consistency verification. The method requires no parameter optimization and uses a small unlabeled reference set for score calibration.

Experiments on COCO and VG-256 demonstrate improvements over global CLIP and several existing training-free methods. Ablation studies show that most of the performance gain comes from regional aggregation, while the calibration components provide smaller and dataset-dependent improvements.

Critical strengths

1. Reasonable insight and well-motivated method design.

The paper addresses a meaningful trade-off between recovering overlooked local objects and avoiding misleading local predictions. The combination of multiple candidate regions, different context granularities, and conditional evidence calibration is reasonably motivated. In particular, using class-wise global-score percentiles to guide local evidence aggregation is an interesting design choice.

2. Clear empirical validation.

The method achieves consistent improvements over global CLIP on both benchmarks. The ablation studies provide a clear breakdown of the contributions from regional aggregation and calibration. The paper also reports cases where semantic verification does not improve performance, which helps establish a balanced interpretation of the results.

Critical weaknesses

1. Limited incremental novelty over closely related methods.

The main insight that local evidence can recover objects overlooked by global CLIP representations has already been explored in prior work. In particular, TagCLIP already performs class-specific region identification followed by crop-based CLIP re-encoding, while PIAA investigates adaptive aggregation of patch-level predictions.

GRADE introduces additional multi-region, multi-granularity, and calibration mechanisms, which provide some methodological novelty. However, most of the reported improvement comes from regional evidence aggregation, while the additional calibration components contribute relatively small gains. This limits the overall significance of the incremental contribution.

2. Limited performance gains relative to the additional inference cost.

Although GRADE substantially improves over global CLIP, its improvements over TagCLIP are relatively modest, approximately 1.60 and 0.98 mAP points on COCO and VG-256, respectively.

Meanwhile, the method requires an average of 64.51 additional regional crop evaluations per image on COCO. The paper does not provide a direct end-to-end latency comparison with the closest baselines. Consequently, it is difficult to assess whether the incremental performance improvements justify the substantially increased inference computation.

Questions for authors

Q1. Given that TagCLIP already performs class-specific crop-based re-identification, could the authors further clarify the practical advantage of GRADE's multi-region and multi-granularity design relative to its additional computational cost? In particular, how should the relatively modest improvement over TagCLIP be interpreted given the large number of additional crop evaluations?

Q2. The improvements from DGC and SCV are relatively small, and SCV slightly reduces mAP on VG-256. Could the authors clarify how robust these calibration gains are to the choice of unlabeled reference samples and calibration settings? This would help assess the general effectiveness of the proposed reliability mechanisms.

4 comments
AuthorsOpenReview ↗

Method design and the accuracy-latency trade-off

Thank you for your feedback. We focused the revision on adaptive single-crop selection and the calibration of local evidence.

W1. We agree that TagCLIP already re-encodes class-specific crops. PIAA studies adaptive aggregation of patch predictions. GRADE selects regions to recover true labels and calibrates local responses to suppress inflated scores for absent classes. The marginal effects of DGC and SCV were measured using identical regions. On VG-256, adding SCV to DGC raised mAP from 40.86% to 41.23%. See Table 2 (p. 9).

W2. The comparison matched input resolution, hardware, and batch size. Relative to matched TagCLIP, GRADE gained 2.81, 2.66, and 10.28 mAP points on COCO, VG-256, and NUS-WIDE, respectively. At batch size 16, GRADE took 12.35, 15.20, and 13.35 ms per image. Matched TagCLIP took 20.65, 40.69, and 21.38 ms per image in the same order. GRADE encodes at most one crop per candidate class. It extracts standard CLIP embeddings from the crops. Matched TagCLIP uses class-specific attention masks to re-encode its crops. SCV reuses existing features and adds about 0.25–0.73 ms per image over the DGC variant. See Figure 6 (p. 10).

Q1. See our responses to W1 and W2.

Q2. See W1 for the SCV gain on VG-256. DGC now uses within-image statistics, removing the need to sample reference images. We tested local fusion weights from 0.375 to 0.625. On all three datasets, mAP differed from the default by at most 0.13 percentage points. See Figure 7 (p. 16).

Our goal is to select informative regions, use their evidence, and suppress local hallucinations within one framework. Thank you again for your feedback.

Thank you for the response and the additional experiments.

In my understanding, this paper is primarily method-driven, with its main contributions lying in the proposed technical design rather than a distinct empirical insight. However, several core components, particularly Multi-Granularity Regional Evidence (MGE), Distribution-Guided Granularity Calibration (DGC), and Semantic Consistency Verification (SCV), appear to have undergone substantial changes in the revised manuscript.

These changes seem to go beyond additional experiments, implementation clarifications, or minor refinements. Given that the methodological design constitutes the central contribution of this work, I am inclined to maintain my original assessment.

AuthorsOpenReview ↗

Core design and scope of the GRADE revision

Thank you for considering our response and the revised manuscript. We agree that the revision made substantive algorithmic changes. Both versions of GRADE address the dual role of local evidence illustrated in Figure 1 of the original submission. Local evidence can recover labels overlooked by global CLIP and can also inflate scores for absent classes. Both versions follow the same basic workflow. They select class-specific regions, calibrate local evidence using global class scores, and check its semantic support before fusion.

The changes address the cost of repeated cropping, the reliance on a sampled reference set, and the modest gains from the original calibration components.

  1. Region selection. The goal in both versions is to capture class-specific evidence at a suitable context scale. The original MGE evaluated multiple regions at several scales. The revision uses global class support to interpolate between two reference boxes. At most one crop is encoded per candidate class.

  2. DGC. Both versions use global class support to regulate the contribution of local evidence. The original DGC used reference-set percentiles to aggregate crop scores across scales. The revised DGC uses within-image statistics of global class scores to weight patch evidence.

  3. SCV. Both versions use semantic support to suppress spurious local responses. The original SCV compared the selected region with related classes when local evidence dominated. The revised SCV uses an auxiliary vocabulary to calibrate patch and crop scores. It also uses the selected crop feature to verify patch evidence through related-class comparisons.

On COCO, the mean number of logical crops fell from 64.51 to 7.74 per image. With the same frozen CLIP ViT-B/16 backbone and evaluation splits, mAP rose from 72.62% to 74.13% on COCO and from 37.63% to 41.23% on VG-256 (Table 1 in each version). Thank you again for your feedback.

AuthorsOpenReview ↗

The research question behind both versions of GRADE

Thank you for explaining your concern about the scope of the revision. We agree that the algorithms changed substantially. Both versions of GRADE are motivated by the observation in Figure 1 of the original submission. Local evidence can recover labels missed by global CLIP. It can also inflate scores for absent classes.

The core idea is to find class-specific local evidence and assess its reliability before fusion\textbf{The core idea is to find class-specific local evidence and assess its reliability before fusion}. In both versions, global class scores guide how much local evidence contributes, and semantic checks help suppress misleading responses. We hope you will consider this joint treatment of label recovery and evidence reliability in your assessment.

Reviewer_1i68OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies training-free zero-shot multi-label recognition with CLIP and focuses on the limitations of relying solely on a global image representation. The authors observe that local crops can recover small or spatially localized objects missed by global CLIP, but can also produce strong responses to absent yet visually or semantically related classes, which they refer to as “local hallucination”. To address this issue, the paper proposes GRADE . GRADE first extracts class-specific connected regions from dense CLIP response maps and evaluates each region under multiple context margins. The resulting regional responses are summarized by the minimum, geometric mean, and maximum across granularities. A Distribution-Guided Granularity Calibration module then uses the percentile of the global class score relative to a 1% unlabeled target-domain reference set to interpolate between these summaries. Finally, when local evidence dominates the global prediction, SCV compares the selected region against semantically neighboring classes using comparative text prompts and attenuates ambiguous local evidence. The final score combines the original global score and the calibrated local contribution. Experiments on COCO and VG-256 show that GRADE improves mAP over global CLIP by 7.57 and 7.16 percentage points, respectively.

Critical strengths
  1. The paper is generally well written. The motivation is easy to understand, the distinction between evidence coverage and local-response reliability is clear, and Figures 1–4 provide useful intuition for the proposed pipeline.
  2. The paper provides a useful empirical observation: local visual evidence is simultaneously beneficial and risky in zero-shot multi-label recognition. Multiple regional hypotheses substantially improve class coverage, while semantic verification can sometimes suppress misleading local responses.
Critical weaknesses
  1. The ablations show that approximately 94–96% of the improvement over global CLIP is already obtained by regional aggregation without the proposed calibration mechanisms. DGC provides only a small additional improvement over simple fixed aggregation, while SCV is nearly neutral overall and slightly harmful on VG-256. The multi-granularity contribution also becomes very small once multiple regions are used.
  2. These gains come at substantial computational cost, with more than 60 regional CLIP evaluations per image on average.
  3. Evaluation is limited to two datasets and one primary CLIP backbone. Since the paper’s gains beyond regional aggregation are small and dataset-dependent, experiments on additional multi-label datasets (such as NUS-WIDE) and stronger/different VLM backbones would be important for establishing generality.
  4. The 1% reference subset is sampled with a single fixed random seed. Since DGC contributes only a few tenths of a percentage point, variance across different reference subsets should be reported before attributing those gains to the proposed calibration mechanism.
Questions for authors
  1. How stable are DGC results across independently sampled 1% reference subsets? Please report mean ± standard deviation over multiple reference-set seeds.
  2. Can the authors report end-to-end inference latency, throughput, and memory for GRADE and the closest training-free baselines under a common hardware/batching setup? An average of 64.51 regional evaluations per COCO image appears substantial.
  3. Could the authors evaluate on additional datasets (e.g., NUS-WIDE) and/or CLIP/VLM backbones? The current evidence is restricted to COCO, VG-256, and primarily CLIP ViT-B/16.
  4. Since arithmetic mean achieves 72.34 mAP versus 72.56 for DGC on COCO, does DGC remain consistently superior to simple fixed aggregation on VG-256 and across other reference seeds? Please provide the corresponding fixed-aggregation control on VG-256.
  5. What is the performance of a computationally cheaper configuration such as R=3 with the best two margins, including the full DGC/SCV pipeline?
  6. TagCLIP uses an appended generic background vocabulary whereas GRADE uses the target vocabulary for its score normalization. Since the scores are softmax-normalized over the vocabulary, can the authors provide a matched-vocabulary TagCLIP comparison to ensure that this implementation difference does not affect AP?
  7. Macro-F1 thresholds are estimated from the entire evaluation-score distribution at a fixed 2% predicted-positive rate. Why was 2% chosen, and how sensitive are conclusions to this choice?
2 comments
AuthorsOpenReview ↗

Calibration controls and expanded evaluation

Thank you for your careful review. We revised GRADE to use adaptive single-crop selection and redesigned DGC and SCV.

W1. The revised GRADE improves mAP over the uncalibrated baseline on all three datasets. The DGC comparison held the regions, patch and crop scores, and fusion weight fixed. Adding DGC's patch weighting raised mAP from 40.11% to 40.86% on VG-256 and from 73.17% to 73.68% on COCO. Adding SCV to DGC raised VG-256 mAP from 40.86% to 41.23%. See Table 2 (p. 9).

W2. Adaptive single-crop selection reduced the mean logical crop count from 64.51 to 7.74 per image on the full COCO set. Both methods were timed with CLIP ViT-B/16 on the same RTX 4090 at batch size 16. GRADE took 12.35, 15.20, and 13.35 ms per image on COCO, VG-256, and NUS-WIDE, respectively. Matched TagCLIP took 20.65, 40.69, and 21.38 ms per image on the same datasets. On COCO, throughput was 80.95 images/s for GRADE and 48.42 images/s for matched TagCLIP. Peak allocated GPU memory was 1.21 and 1.20 GiB, respectively. Timing starts from preprocessed CPU tensors and covers transfer, region selection, cropping, encoding, calibration, and fusion. See Table 5 (p. 17).

W3. We added an aligned NUS-WIDE evaluation with 83,898 records containing at least one target label. GRADE achieved 54.05% mAP and 34.21% Macro-F1. Global CLIP achieved 50.60% mAP. CLIP ViT-B/16 was used for a matched comparison with TagCLIP. Further work should evaluate other backbones. See Table 1 (p. 8).

W4. DGC now standardizes log-transformed global class scores within each image to weight patch evidence. This removes the dependence on a 1% reference subset. See Equations (16)–(18) on p. 6.

Q1. See our response to W4.

Q2. See our response to W2.

Q3. See our response to W3.

Q4. See W1 for the fixed aggregation comparison and W4 for reference-set dependence.

Q5. The revised single-crop configuration retains DGC and SCV. Its mAP is 74.13% on COCO, 41.23% on VG-256, and 54.05% on NUS-WIDE. We also evaluated an eight-crop cap per image using saved responses. The mAP losses were 0.24, 0.16, and less than 0.01 percentage points, respectively. See Figure 7 (p. 16).

Q6. On the full aligned evaluation sets, the matched TagCLIP comparison held the target prototypes fixed. Replacing the fixed auxiliary vocabulary with the SCV auxiliary vocabulary raised mAP by 0.01–0.18 percentage points. The fixed auxiliary vocabulary improved mAP over target-only scoring on all three fixed subsets. See Table 6 (p. 17).

Q7. The 2% nominal predicted-positive rate was chosen empirically as a common operating point. Global CLIP reached near-peak Macro-F1 at this rate on COCO and VG-256. The threshold for each class is the 98th percentile of its evaluation scores. We tested 19 nominal rates from 0.25% to 20%. GRADE outperformed global CLIP at every sampled rate on all three datasets. See Figure 8 (p. 16).

Our goal is to select informative regions, use their evidence, and suppress local hallucinations within one framework. Thank you again for your feedback.

AuthorsOpenReview ↗

Evaluating the dual role of local evidence

Thank you for highlighting the dual role of local evidence in your review. Studying how it recovers missed labels and raises scores for absent classes gives us a basis for designing calibration in multi-label image recognition. Table 2 holds the selected regions fixed to assess what calibration contributes. We hope you will take this perspective and the additional evidence into account when revisiting your assessment.

Reviewer_nTTDOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes training-free zero-shot multi-label recognition (MLR) method for addressing discrepancy on global and local perception ability of CLIP. The method includes four stages: (1) Multi-Region Extraction for selecting candidate regions (2) Multi-Granularity Evidence (MGE) for obtaining diverse local information based on CLIP score (3) Distribution-Guided Calibration (DGC) for scoring the local evidence based on global evidence (4) Semantic Consistency Verification (SCV) for weakening ambiguous local responses within image. Without training, proposed method achieved higher performance than training-based method such as DualCoOp on two benchmark datasets.

Critical strengths
  1. The paper handles the limitations about perception ability of CLIP for MLR task. In particular, CLIP is weak at capturing small objects through global CLIP score and can be confuse the objects from local crops. The authors address these issues through several aggregation and verification stages to balance local evidence.

  2. The proposed GRADE achieves the best performance on two benchmark datasets without training.

Critical weaknesses
  1. Lack of the inference time and computation costs comparison. The proposed method consists of multiple procedures and requires processing a large number of local crops. Thus, computation cost comparison should be reported.

  2. What if there are a few large and clearly visible objects within images? In such cases, additional local evidence may not be necessary. Without considering whether it is needed, which may lead to unnecessary computational costs.

  3. The performance improvements over existing methods are marginal on the benchmark datasets, although the proposed method requires multiple procedures. Thus, the novelty and effectiveness of the proposed method are not clear.

Questions for authors

I am wondering the inference time comparison with existing methods and the novelty of GRADE. Moreover, is it possible to reduce unnecessary multiple processing through adaptive inference?

4 comments
AuthorsOpenReview ↗

Inference cost and the benefit of local crops

Thank you for your careful review. We replaced repeated cropping with adaptive single-crop selection and streamlined the method description.

W1. We agree that inference cost should be measured directly. GRADE now encodes at most one crop per candidate class. On the full COCO evaluation set, the mean logical crop count fell from 64.51 to 7.74 per image. Both methods were timed on the same RTX 4090 at batch size 16. On COCO, GRADE and matched TagCLIP took 12.35 and 20.65 ms per image, respectively. See Figure 6 (p. 10).

W2. Your comment motivated us to define a subset with only a few large, clearly visible objects per image. The main comparison covered 36 classes with sufficient support. Global CLIP reached 94.30% mAP. GRADE reached 95.78% without crops and 95.96% with crops. Across all 74 evaluable classes in this subset, removing crops lowered mAP by 1.15 percentage points. The loss was 2.39 points in the mixed-size subset. In that subset, recall for labels represented only by small instances fell from 61.25% to 57.73%. The mAP loss on the full COCO set was larger than on the subset with few large objects. GRADE therefore retains adaptive single-crop selection. Further work should clarify when crop encoding can be skipped. See Table 8 (p. 18).

W3. The revision focuses on region selection and evidence calibration under a fixed single-crop budget. CGE and AGS select regions using contrast, coverage, and global class support. DGC weights patch evidence. SCV calibrates patch and crop responses. The regions and raw visual responses were held fixed. Together, DGC and SCV raised mAP by 0.97–1.24 percentage points over the uncalibrated baseline across the three datasets. See Table 2 (p. 9).

Q1. See W1 for inference time and W3 for the contribution of the revised design. We also tested an early-exit rule based on first-pass global and patch scores. At batch size 16, latency fell from 11.25 to 10.67 ms per image on the subset with few large objects. The mAP drop was 0.05 percentage points. Latency fell by 2.1% on a sample from the full COCO set. See Table 9 (p. 19).

Our goal is to select informative regions, use their evidence, and suppress local hallucinations within one framework. Thank you again for your feedback.

AuthorsOpenReview ↗

When local crops help

Thank you for asking when local crops are useful. Your question led to the analysis in Appendix D.6, where cropping yielded smaller gains for images with a few large objects. Studying label recovery alongside inflated scores for absent classes can help determine when local evidence is worth using in multi-label image recognition. We would be grateful if you could reconsider the paper in light of our response and these results.

Thank you for the detailed rebuttal. I appreciate the improved accuracy and inference efficiency, particularly compared with TagCLIP.

However, three concerns remain.

Q1. Substantial methodological changes

As discussed by Reviewer zCXH, the authors substantially redesigned the core methodology during rebuttal, replacing multi-region, multi-granularity aggregation with adaptive single-crop selection and significantly modifying DGC and SCV. The authors explicitly acknowledged these changes.

Although these modifications improve accuracy and efficiency, they go beyond additional experiments or minor refinements and substantially alter the technical contributions of the original submission. This raises concerns about evaluating the revised methodology within the scope of the original review.

Q2. Limited contributions of individual components

Table 2 shows that CGE+AGS alone achieves 73.17, 40.11, and 52.82 mAP on COCO, VG-256, and NUS-WIDE, respectively. Adding DGC and SCV improves performance by only 0.96–1.23 percentage points.

Furthermore, adaptive granularity selection provides only marginal improvements over the best individual reference (+0.14 and +0.13 mAP on COCO and NUS-WIDE, respectively), while slightly underperforming the contrast reference on VG-256 (−0.03 mAP). Thus, the methodological novelty and necessity of the individual components remain insufficiently justified.

Q3. Insufficient table and figure captions

Several table and figure captions lack sufficient explanations of the experimental settings, comparison conditions, and reported results. More self-contained captions would improve the clarity and readability of the manuscript.

Overall assessment

While the rebuttal substantially addresses my concerns regarding inference efficiency, the extensive methodological changes and limited incremental contributions leave my concerns regarding novelty unresolved. Therefore, I maintain my original rating of weak rejection.

AuthorsOpenReview ↗

Method revisions, component contributions, and captions

Thank you for taking the time to read our rebuttal and explain your remaining concerns.

Q1. We agree that the revision made substantive methodological changes. The cost of repeated encoding motivated the move to at most one crop per candidate class. In the revised method, DGC uses within-image statistics of global class scores to weight patch evidence. SCV reuses existing features to check patch and crop responses. The two versions share the same design goals\textbf{The two versions share the same design goals}. Region selection seeks evidence for labels overlooked by global CLIP. Global class support guides how much local evidence contributes, and semantic verification suppresses misleading local responses. Figure 1 of the original submission (p. 2) illustrates the dual role of local evidence that motivates this design.

Q2. The better single reference varies across datasets. AGS combines contrast and coverage under the same crop\textbf{The better single reference varies across datasets. AGS combines contrast and coverage under the same crop} budget\textbf{budget}. It achieves the highest mAP on COCO and NUS-WIDE and trails the best reference by 0.03 points on VG-256 (Table 2, p. 9). With regions and raw visual scores held fixed, DGC and SCV each improve mAP when added to the other on all three datasets. Together, they add about one mAP point per dataset.

Figure 4 (p. 9) shows a concrete effect of SCV. A striped region gives the absent class “zebra” a high patch score. SCV uses the selected crop to check that response, moving “zebra” from fourth to 30th. This would remove a false tag in a system that uses the top five predictions.

Q3. We revised the caption of Table 2 (p. 9) to identify the adaptive comparison, state the shared crop budget, and explain which inputs were held fixed in the calibration ablation. The caption of Figure 5 (p. 9) now defines the base configuration and the class-wise AP differences. The caption of Figure 6 (p. 10) now distinguishes mAP on the full evaluation sets from timing on the same 512 images per dataset and specifies the shared timing setup.

The central contribution of GRADE is to address both what local evidence can recover and how it can mislead recognition. The selected crop provides class-specific evidence and helps check the patch responses that guided its selection. Treating label recovery and local hallucinations together offers a way to study when local evidence can be trusted in training-free multi-label\textbf{Treating label recovery and local hallucinations together offers a way to study when local evidence can be trusted in training-free multi-label} image recognition\textbf{image recognition}. We hope you will consider this contribution when revisiting your assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.