- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes GRADE, a training-free framework for zero-shot multi-label image recognition. The authors observe that global CLIP representations may overlook small or localized objects, while local crops can introduce misleading class responses.
To address these issues, GRADE combines class-specific multi-region extraction, multi-granularity evidence aggregation, distribution-guided calibration, and semantic consistency verification. The method requires no parameter optimization and uses a small unlabeled reference set for score calibration.
Experiments on COCO and VG-256 demonstrate improvements over global CLIP and several existing training-free methods. Ablation studies show that most of the performance gain comes from regional aggregation, while the calibration components provide smaller and dataset-dependent improvements.
Critical strengths
1. Reasonable insight and well-motivated method design.
The paper addresses a meaningful trade-off between recovering overlooked local objects and avoiding misleading local predictions. The combination of multiple candidate regions, different context granularities, and conditional evidence calibration is reasonably motivated. In particular, using class-wise global-score percentiles to guide local evidence aggregation is an interesting design choice.
2. Clear empirical validation.
The method achieves consistent improvements over global CLIP on both benchmarks. The ablation studies provide a clear breakdown of the contributions from regional aggregation and calibration. The paper also reports cases where semantic verification does not improve performance, which helps establish a balanced interpretation of the results.
Critical weaknesses
1. Limited incremental novelty over closely related methods.
The main insight that local evidence can recover objects overlooked by global CLIP representations has already been explored in prior work. In particular, TagCLIP already performs class-specific region identification followed by crop-based CLIP re-encoding, while PIAA investigates adaptive aggregation of patch-level predictions.
GRADE introduces additional multi-region, multi-granularity, and calibration mechanisms, which provide some methodological novelty. However, most of the reported improvement comes from regional evidence aggregation, while the additional calibration components contribute relatively small gains. This limits the overall significance of the incremental contribution.
2. Limited performance gains relative to the additional inference cost.
Although GRADE substantially improves over global CLIP, its improvements over TagCLIP are relatively modest, approximately 1.60 and 0.98 mAP points on COCO and VG-256, respectively.
Meanwhile, the method requires an average of 64.51 additional regional crop evaluations per image on COCO. The paper does not provide a direct end-to-end latency comparison with the closest baselines. Consequently, it is difficult to assess whether the incremental performance improvements justify the substantially increased inference computation.
Questions for authors
Q1. Given that TagCLIP already performs class-specific crop-based re-identification, could the authors further clarify the practical advantage of GRADE's multi-region and multi-granularity design relative to its additional computational cost? In particular, how should the relatively modest improvement over TagCLIP be interpreted given the large number of additional crop evaluations?
Q2. The improvements from DGC and SCV are relatively small, and SCV slightly reduces mAP on VG-256. Could the authors clarify how robust these calibration gains are to the choice of unlabeled reference samples and calibration settings? This would help assess the general effectiveness of the proposed reliability mechanisms.
4 comments
Thank you for the response and the additional experiments.
In my understanding, this paper is primarily method-driven, with its main contributions lying in the proposed technical design rather than a distinct empirical insight. However, several core components, particularly Multi-Granularity Regional Evidence (MGE), Distribution-Guided Granularity Calibration (DGC), and Semantic Consistency Verification (SCV), appear to have undergone substantial changes in the revised manuscript.
These changes seem to go beyond additional experiments, implementation clarifications, or minor refinements. Given that the methodological design constitutes the central contribution of this work, I am inclined to maintain my original assessment.
Core design and scope of the GRADE revision
Thank you for considering our response and the revised manuscript. We agree that the revision made substantive algorithmic changes. Both versions of GRADE address the dual role of local evidence illustrated in Figure 1 of the original submission. Local evidence can recover labels overlooked by global CLIP and can also inflate scores for absent classes. Both versions follow the same basic workflow. They select class-specific regions, calibrate local evidence using global class scores, and check its semantic support before fusion.
The changes address the cost of repeated cropping, the reliance on a sampled reference set, and the modest gains from the original calibration components.
-
Region selection. The goal in both versions is to capture class-specific evidence at a suitable context scale. The original MGE evaluated multiple regions at several scales. The revision uses global class support to interpolate between two reference boxes. At most one crop is encoded per candidate class.
-
DGC. Both versions use global class support to regulate the contribution of local evidence. The original DGC used reference-set percentiles to aggregate crop scores across scales. The revised DGC uses within-image statistics of global class scores to weight patch evidence.
-
SCV. Both versions use semantic support to suppress spurious local responses. The original SCV compared the selected region with related classes when local evidence dominated. The revised SCV uses an auxiliary vocabulary to calibrate patch and crop scores. It also uses the selected crop feature to verify patch evidence through related-class comparisons.
On COCO, the mean number of logical crops fell from 64.51 to 7.74 per image. With the same frozen CLIP ViT-B/16 backbone and evaluation splits, mAP rose from 72.62% to 74.13% on COCO and from 37.63% to 41.23% on VG-256 (Table 1 in each version). Thank you again for your feedback.
The research question behind both versions of GRADE
Thank you for explaining your concern about the scope of the revision. We agree that the algorithms changed substantially. Both versions of GRADE are motivated by the observation in Figure 1 of the original submission. Local evidence can recover labels missed by global CLIP. It can also inflate scores for absent classes.
. In both versions, global class scores guide how much local evidence contributes, and semantic checks help suppress misleading responses. We hope you will consider this joint treatment of label recovery and evidence reliability in your assessment.
Method design and the accuracy-latency trade-off
Thank you for your feedback. We focused the revision on adaptive single-crop selection and the calibration of local evidence.
W1. We agree that TagCLIP already re-encodes class-specific crops. PIAA studies adaptive aggregation of patch predictions. GRADE selects regions to recover true labels and calibrates local responses to suppress inflated scores for absent classes. The marginal effects of DGC and SCV were measured using identical regions. On VG-256, adding SCV to DGC raised mAP from 40.86% to 41.23%. See Table 2 (p. 9).
W2. The comparison matched input resolution, hardware, and batch size. Relative to matched TagCLIP, GRADE gained 2.81, 2.66, and 10.28 mAP points on COCO, VG-256, and NUS-WIDE, respectively. At batch size 16, GRADE took 12.35, 15.20, and 13.35 ms per image. Matched TagCLIP took 20.65, 40.69, and 21.38 ms per image in the same order. GRADE encodes at most one crop per candidate class. It extracts standard CLIP embeddings from the crops. Matched TagCLIP uses class-specific attention masks to re-encode its crops. SCV reuses existing features and adds about 0.25–0.73 ms per image over the DGC variant. See Figure 6 (p. 10).
Q1. See our responses to W1 and W2.
Q2. See W1 for the SCV gain on VG-256. DGC now uses within-image statistics, removing the need to sample reference images. We tested local fusion weights from 0.375 to 0.625. On all three datasets, mAP differed from the default by at most 0.13 percentage points. See Figure 7 (p. 16).
Our goal is to select informative regions, use their evidence, and suppress local hallucinations within one framework. Thank you again for your feedback.