- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes an efficient and effective inference framework for zero-shot multi-label recognition (MLR) based on MLLM. In particular, Structured Parallel Querying (SPQ) introduces a shared prefix and independent category predictions by modifying the attention process and positional encoding. Without autoregressive generation, this method obtains class-wise yes/no scores by comparing the corresponding logits for each candidate class. These processes are intuitive and effective. The proposed method achieves competitive results on four benchmark datasets and fast inference.
Critical strengths
-
The problem and method is straightforward and intuitive. It addresses the limitations of MLLMs in terms of causal attention and heavy inference costs.
-
The paper is easy to understand and follow.
-
The proposed method shows good recognition performance and significantly improves inference costs compared to independent binary querying.
Critical weaknesses
- Lack of failure cases analysis. Failure case analysis is needed to better understand the limitations of the proposed method.
Questions for authors
Please, provide the failure cases analysis to better understand the limitations of the proposed method.
This method is based on the modifying attention process of MLLM. Is it possible to extend the proposed method to recognize fine-grained categories, such as red cars and blue cars?
4 comments
Section 4.6 and Figure 7 present the fine-grained recognition results, with complete Top-1 accuracy values in Appendix H (Table 12). Appendix I (Figure 11) provides the class-wise COCO error analysis, while the full FP/FN counts and object-area statistics are included in response Table 2. Additional small-object recognition examples are shown in Figure 3(b) and Figure 10.
Thank you for author response and my concern is resolved. I will maintain my current positive score.
Thank You for Your Positive Feedback
We sincerely thank the reviewer for the positive feedback and for reconsidering our response. We are pleased that our clarifications and additional analyses have addressed your concerns. We greatly appreciate your constructive comments and continued support of our work.
We thank the reviewer for the positive assessment of SPQ and for requesting failure-case analysis and clarification of its applicability to fine-grained categories. We address both points with additional experiments and discuss the resulting limitations.
1. Fine-grained category recognition
SPQ is not restricted to atomic category names: attribute-qualified expressions such as red car and blue car can be formulated as separate class-presence queries without modifying the architecture. Correctly distinguishing such expressions, however, depends on the pretrained MLLM's ability to bind the visual attribute to the intended object. To test fine-grained recognition more broadly, we evaluate CUB-200, Flowers102, Food101, and OxfordPets without task-specific training. Each image is assigned the highest-scoring category (Top-1 accuracy). We compare SPQ with candidate-guided generation (CGG), which receives the full candidate list and generates the predicted category name, and include three zero-shot CLIP models as references.
Table 1. Fine-grained zero-shot recognition (Top-1 accuracy, %). All values are reproduced in manuscript Appendix H, Table 12.
SPQ outperforms CGG in 15 of 16 backbone–dataset settings. The largest improvements include Food101 with Qwen-7B (32.17% to 82.29%, +50.12 points) and OxfordPets with Qwen-3B (39.71% to 84.38%, +44.67 points). Nevertheless, the improvement is backbone-dependent: InternVL slightly underperforms CGG on CUB-200 (8.77% versus 9.29%), and CLIP ViT-L/14 retains the highest Top-1 accuracy on all four benchmarks. These results establish the utility of structured scoring for fine-grained categories.
2. Failure-case analysis
We evaluate Qwen2.5-VL-3B on all 17 unseen COCO categories with a fixed Yes/No score threshold of 0.5. Table 2 reports the mean annotated object area, false-positive (FP) and false-negative (FN) counts, and false-positive and false-negative rates (FPR and FNR). FPR is calculated over negative instances and FNR over positive instances for each category.
Table 2. Complete class-wise COCO error statistics at threshold 0.5. The manuscript visualizes the 17-class FNR/FPR results in Appendix I, Figure 11.
The errors are markedly class-dependent. FNR ranges from 1.56% for airplane to 52.79% for knife, with cup also reaching 51.47%. By contrast, FPR remains below 4% for every class, peaking at 3.69% for tie. Mean object area shows an overall negative association with FNR across these 17 categories (Spearman ρ ≈ −0.60), but the relationship is not uniform: skateboard has only 2.1% mean area yet 2.94% FNR, whereas couch has 15.5% mean area and 21.25% FNR. The area–FPR association is weak (ρ ≈ −0.20). These descriptive associations do not establish object size as the cause of the errors.
Importantly, the high thresholded FNRs for knife and cup do not contradict their relative class-wise AP advantages over CLIP discussed in the paper. AP measures how well scores rank images for a given category, whereas FNR evaluates missed positives at a fixed decision threshold. Thus, the analysis identifies a concrete limitation—uneven category-level recall at threshold 0.5—without attributing it to an unverified single mechanism.