acceptodds
Under review as a conference paper at ICLR 2027

One Image, Parallel Decisions: Structured Querying for Zero-Shot Multi-Label Recognition

Abstract

Contrastive vision–language models such as CLIP have been widely adopted for zero-shot multi-label recognition, while generative multimodal large language models (MLLMs) have received comparatively less attention in this setting. Applying MLLMs to this task requires an interface that produces scores for multiple non-exclusive categories from generative predictions. Direct label generation does not readily provide a score for every candidate class, whereas evaluating class-specific questions separately repeats shared context computation. We propose Structured Parallel Querying (SPQ), a training-free method that organizes class-presence questions as independent suffixes over a shared visual and instructional prefix. Block-isolated attention prevents information exchange between class suffixes, while aligned positional encodings preserve the context of standalone queries. SPQ reads Yes/No logits at all class-specific decision positions in a single joint forward pass, producing multi-label scores without autoregressive answer generation. Experiments with four MLLM configurations spanning Qwen2.5-VL, LLaVA, and InternVL demonstrate the applicability of SPQ across model families and scales. A direct comparison with independent binary querying further shows that joint evaluation can preserve recognition performance. These findings support structured parallel querying as a practical approach to zero-shot multi-label recognition without task-specific training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-label classification, Zero-shot learning, Multimodal large language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_prUbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an efficient and effective inference framework for zero-shot multi-label recognition (MLR) based on MLLM. In particular, Structured Parallel Querying (SPQ) introduces a shared prefix and independent category predictions by modifying the attention process and positional encoding. Without autoregressive generation, this method obtains class-wise yes/no scores by comparing the corresponding logits for each candidate class. These processes are intuitive and effective. The proposed method achieves competitive results on four benchmark datasets and fast inference.

Critical strengths
  1. The problem and method is straightforward and intuitive. It addresses the limitations of MLLMs in terms of causal attention and heavy inference costs.

  2. The paper is easy to understand and follow.

  3. The proposed method shows good recognition performance and significantly improves inference costs compared to independent binary querying.

Critical weaknesses
  1. Lack of failure cases analysis. Failure case analysis is needed to better understand the limitations of the proposed method.
Questions for authors

Please, provide the failure cases analysis to better understand the limitations of the proposed method.

This method is based on the modifying attention process of MLLM. Is it possible to extend the proposed method to recognize fine-grained categories, such as red cars and blue cars?

4 comments

We thank the reviewer for the positive assessment of SPQ and for requesting failure-case analysis and clarification of its applicability to fine-grained categories. We address both points with additional experiments and discuss the resulting limitations.

1. Fine-grained category recognition

SPQ is not restricted to atomic category names: attribute-qualified expressions such as red car and blue car can be formulated as separate class-presence queries without modifying the architecture. Correctly distinguishing such expressions, however, depends on the pretrained MLLM's ability to bind the visual attribute to the intended object. To test fine-grained recognition more broadly, we evaluate CUB-200, Flowers102, Food101, and OxfordPets without task-specific training. Each image is assigned the highest-scoring category (Top-1 accuracy). We compare SPQ with candidate-guided generation (CGG), which receives the full candidate list and generates the predicted category name, and include three zero-shot CLIP models as references.

Table 1. Fine-grained zero-shot recognition (Top-1 accuracy, %). All values are reproduced in manuscript Appendix H, Table 12.

BackboneMethodCUB-200Flowers102Food101OxfordPets
CLIP ViT-B/32CLIP49.6061.7581.0079.89
CLIP ViT-B/16CLIP53.3864.4887.0184.46
CLIP ViT-L/14CLIP61.3473.1391.4290.24
Qwen2.5-VL-3BSPQ51.2664.9173.6884.38
Qwen2.5-VL-3BCGG19.8750.9538.2439.71
Qwen2.5-VL-7BSPQ50.3170.8182.2987.46
Qwen2.5-VL-7BCGG22.3930.1532.1745.27
InternVL3.5-4BSPQ8.7719.4740.0030.44
InternVL3.5-4BCGG9.2916.0528.8025.95
LLaVA-1.5-7BSPQ10.7525.4258.7928.81
LLaVA-1.5-7BCGG5.6110.4919.9927.34

SPQ outperforms CGG in 15 of 16 backbone–dataset settings. The largest improvements include Food101 with Qwen-7B (32.17% to 82.29%, +50.12 points) and OxfordPets with Qwen-3B (39.71% to 84.38%, +44.67 points). Nevertheless, the improvement is backbone-dependent: InternVL slightly underperforms CGG on CUB-200 (8.77% versus 9.29%), and CLIP ViT-L/14 retains the highest Top-1 accuracy on all four benchmarks. These results establish the utility of structured scoring for fine-grained categories.

2. Failure-case analysis

We evaluate Qwen2.5-VL-3B on all 17 unseen COCO categories with a fixed Yes/No score threshold of 0.5. Table 2 reports the mean annotated object area, false-positive (FP) and false-negative (FN) counts, and false-positive and false-negative rates (FPR and FNR). FPR is calculated over negative instances and FNR over positive instances for each category.

Table 2. Complete class-wise COCO error statistics at threshold 0.5. The manuscript visualizes the 17-class FNR/FPR results in Appendix I, Figure 11.

CategoryMean area (%)FPFPR (%)FNFNR (%)
knife1.0251.0012352.79
skateboard2.1180.7052.94
snowboard2.770.272223.91
tie3.5933.694822.97
cup3.7582.6126351.47
scissors4.7180.671424.14
sink5.1140.576022.73
keyboard7.6672.571713.60
umbrella9.2160.644418.49
airplane10.0100.3821.56
dog12.0391.542311.92
cow15.240.151513.27
couch15.5421.695121.25
cake16.7351.373018.07
cat18.4200.80125.15
bus23.6150.592412.18
elephant27.6100.3821.60

The errors are markedly class-dependent. FNR ranges from 1.56% for airplane to 52.79% for knife, with cup also reaching 51.47%. By contrast, FPR remains below 4% for every class, peaking at 3.69% for tie. Mean object area shows an overall negative association with FNR across these 17 categories (Spearman ρ ≈ −0.60), but the relationship is not uniform: skateboard has only 2.1% mean area yet 2.94% FNR, whereas couch has 15.5% mean area and 21.25% FNR. The area–FPR association is weak (ρ ≈ −0.20). These descriptive associations do not establish object size as the cause of the errors.

Importantly, the high thresholded FNRs for knife and cup do not contradict their relative class-wise AP advantages over CLIP discussed in the paper. AP measures how well scores rank images for a given category, whereas FNR evaluates missed positives at a fixed decision threshold. Thus, the analysis identifies a concrete limitation—uneven category-level recall at threshold 0.5—without attributing it to an unverified single mechanism.

Section 4.6 and Figure 7 present the fine-grained recognition results, with complete Top-1 accuracy values in Appendix H (Table 12). Appendix I (Figure 11) provides the class-wise COCO error analysis, while the full FP/FN counts and object-area statistics are included in response Table 2. Additional small-object recognition examples are shown in Figure 3(b) and Figure 10.

Thank you for author response and my concern is resolved. I will maintain my current positive score.

AuthorsOpenReview ↗

Thank You for Your Positive Feedback

We sincerely thank the reviewer for the positive feedback and for reconsidering our response. We are pleased that our clarifications and additional analyses have addressed your concerns. We greatly appreciate your constructive comments and continued support of our work.

Reviewer_dJRgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper proposes an inference method, SPQ, that allows a decoder-style VLM to perform multilabel recognition by processing the image and instruction prefix once and processing the class tokens in parallel without them influencing each other, so that the VLM behaves as if it were being independently queried about each class. This is done through attention masks that keep classes from interacting with each other and position embeddings that put each class immediately after the question. SPQ performs competitively against CLIP-based methods on multiple benchmarks and is much faster than independent binary queries and candidate-list methods while matching accuracy.

Critical strengths
  1. The method is straightforward and appears to work well, leading to a substantial speedup over independent binary queries.

  2. The paper is well-written and easy to read.

Critical weaknesses
  1. The novelty of the contribution is not clear. As mentioned in Section 2.3, (Wang et al., 2023) previously introduced the idea of using attention masking and repeated position embeddings to keep multiple-choice options from attending to each other. MLR could be viewed as a variant of multiple-choice where more than one option can be chosen. Lines 159-161 seem to suggest that SPQ is novel because of its multimodal setting. Could the authors elaborate on this? What new challenges does the visual modality introduce that are relevant to this approach?

  2. SPQ is shown to be faster than independent binary queries and candidate-list generation while preserving accuracy. However, there is no comparison against any other decoder-style VLM baselines, such as autoregressive querying, to check whether SPQ gets better accuracy than methods with similar runtime. Table 1 only compares to CLIP-based methods, which is not an apples-to-apples comparison due to different model backbones. One possible autoregressive baseline would be for the VLM to output a list of classes and Yes/No answers e.g. "cat : Yes, dog : No, penguin : Yes,...". This format could be enforced via constrained decoding as the variable tokens ("Yes/No") are fixed length. Then we could establish whether inter-class dependencies are helpful or harmful. The ablation in Fig. 3(a) might suggest that inter-class dependencies are harmful, but that could be due to prompt ambiguity (e.g. "Is the following category present? cat dog penguin" doesn't unambiguously imply that we only want an answer for the penguin class). Outputting only the predicted present classes would be another possible autoregressive baseline, even if predicted negative class probabilities are not available.

Questions for authors

See weaknesses.

4 comments

We thank the reviewer for the careful assessment. We clarify the contribution relative to prior structured inference methods and present additional autoregressive comparisons, a controlled answer intervention, and matched efficiency measurements.

1. Contribution and relationship to UniMC

We acknowledge that option-specific attention isolation and positional reuse have precedents, including UniMC (Wang et al., 2023). As discussed in Related Work, UniMC addresses multiple-choice natural language understanding, whereas SPQ targets non-exclusive visual categories with pretrained causal MLLMs. SPQ reuses a shared multimodal prefix, isolates the class-query suffixes, preserves their standalone positional contexts, and reads the Yes/No logits of every category in one forward pass. The contribution is the resulting complete, efficient multimodal class-scoring interface, rather than a new generic attention primitive. Independent querying removes dependence on earlier generated class answers; it does not preclude useful semantic relations among categories. The same interface also supports seen-class adaptation and zero-shot transfer: language-only adaptation yields the highest unseen-class mAP in five of eight backbone–dataset settings, with joint adaptation leading in the remaining three.

2. Autoregressive baselines: recognition and order sensitivity

We compare SPQ with two alternatives on Qwen2.5-VL-3B. Constrained autoregressive generation (AR) receives an ordered candidate list and predicts a Yes/No answer for each category in sequence. Later decisions condition on earlier generated answers; the two-answer-normalized Yes probabilities supply class scores for mAP and F1@kk. Candidate-guided generation (CGG) receives the same complete candidate list and generates the names of predicted positive categories rather than a per-category answer. Its output is evaluated as a set using Micro-F1. We evaluate three candidate permutations on VOC, COCO, and NUS-WIDE using fixed images, candidate sets, and model parameters.

Table 1. Autoregressive recognition under three candidate orders (%). CGG columns report set-level Micro-F1 at the corresponding order; all remaining columns report SPQ/AR class-scoring metrics. The order index in each AR row also identifies the corresponding CGG candidate order.

MethodVOC mAPVOC F1@1VOC F1@2VOC CGG Micro-F1COCO mAPCOCO F1@3COCO F1@5COCO CGG Micro-F1NUS mAPNUS F1@3NUS F1@5NUS CGG Micro-F1
SPQ94.4551.1731.01—88.7853.8237.83—49.2953.4948.98—
AR-192.9546.7426.6169.8581.8449.9632.7978.4346.2649.7443.6118.27
AR-294.4748.6928.5772.5187.7251.3935.7381.5044.4747.6941.5722.54
AR-391.1745.5725.6061.8285.0450.4734.1166.9241.1745.2943.1915.89

Table 1 shows that SPQ exceeds AR in all evaluated F1@kk comparisons. It also exceeds the best tested AR mAP on COCO and NUS-WIDE by 1.06 and 3.03 percentage points, respectively; on VOC, the best AR order gives comparable mAP but lower F1. Across the three orders, AR mAP varies by 3.30/5.88/5.09 points on VOC/COCO/NUS-WIDE, whereas SPQ remains unchanged. CGG Micro-F1 varies by 10.69/14.58/6.65 points, respectively. Although CGG's set-level Micro-F1 is not directly comparable to the ranking-based F1@kk of SPQ and AR, both generative alternatives show pronounced order dependence.

3. Fixed-order intervention and interpretation of Fig. 3(a)

To distinguish sensitivity to the candidate order from dependence on answer history, we fix one category order with sink first and the same 16 categories following it. For ten images where sink is present, we force its answer from Yes to No; for ten other images where it is absent, we force No to Yes. In each paired comparison, the image, class order, and model parameters are unchanged. We examine the subsequent Yes probabilities under the intervention.

Table 2. Fixed-order answer intervention on the 16 subsequent COCO classes. Entries show mean Yes probability before → after intervention (%), followed by the change in percentage points; each direction uses ten images.

Subsequent classSink present: Yes → NoSink absent: No → Yes
scissors47.7 → 21.2 (−26.5)13.8 → 39.3 (+25.5)
cake35.3 → 23.7 (−11.6)14.1 → 33.8 (+19.7)
cat25.7 → 20.8 (−4.9)20.8 → 29.3 (+8.5)
cup38.1 → 35.9 (−2.2)13.1 → 20.3 (+7.2)
bus1.4 → 4.4 (+3.0)22.3 → 24.1 (+1.8)
cow6.7 → 5.4 (−1.3)29.3 → 30.9 (+1.6)
dog6.6 → 5.4 (−1.2)32.4 → 30.9 (−1.5)
knife28.8 → 24.2 (−4.6)19.4 → 28.6 (+9.2)
couch4.4 → 3.5 (−0.9)17.0 → 21.5 (+4.5)
airplane4.3 → 4.6 (+0.3)16.5 → 19.8 (+3.3)
umbrella6.3 → 5.1 (−1.2)27.5 → 31.6 (+4.1)
tie5.3 → 3.1 (−2.2)8.4 → 12.0 (+3.6)
skateboard6.5 → 2.4 (−4.1)10.0 → 15.6 (+5.6)
elephant5.4 → 4.5 (−0.9)19.5 → 23.2 (+3.7)
snowboard7.6 → 1.0 (−6.6)6.9 → 10.7 (+3.8)
keyboard5.6 → 5.8 (+0.2)15.8 → 18.8 (+3.0)
Mean shift−4.04 pp+6.48 pp

The mean downstream score shifts are −4.04 and +6.48 points for the two interventions. For scissors, the changes reach −26.5 and +25.5 points (Table 2). This demonstrates dependence on previous generated answers even with the candidate order fixed. The magnitudes vary by class; the intervention does not imply that all category relations are detrimental.

Figure 3(a) addresses a distinct issue: the interaction between isolation and positional alignment. Removing isolation while retaining reset positions lowers Qwen-3B/7B COCO mAP from 88.78%/91.30% to 13.16%/12.53%, whereas removing both largely restores it to 88.70%/91.23%. Reusing positions without the corresponding attention isolation breaks the standalone-query context. This does not establish that category relations themselves are inherently harmful.

4. Latency and memory under matched autoregressive baselines

Table 3. Same-backbone efficiency of SPQ and autoregressive methods on VOC and COCO. Peak memory denotes PyTorch allocated GPU memory.

DatasetMethodLatency (ms)Peak memory (MiB)Relative latency
VOC (4)SPQ65.807,2981.00×
VOC (4)AR216.597,2983.29×
VOC (4)CGG123.777,3031.88×
COCO (17)SPQ67.597,3011.00×
COCO (17)AR669.197,2989.90×
COCO (17)CGG167.567,3052.48×

Table 3 reports matched per-image measurements. SPQ is 3.29×/9.90× faster than AR and 1.88×/2.48× faster than CGG on VOC/COCO, with comparable peak allocated memory. Autoregressive methods reuse cached states but still require serial output-token decoding; SPQ obtains all class scores without generating answer tokens.

We have clarified the contribution relative to UniMC in Section 2.3 and the SPQ design in Sections 3.2–3.5. Section 4.3 and Table 2 present the autoregressive comparisons and order-sensitivity analysis, with additional results in Appendix F.1 (Table 5). The answer-history intervention is reported in Appendix F.2 (Table 6), alongside the interpretation of Figure 3(a). Section 4.4 and Figure 4 present the updated efficiency results, supplemented by Appendix A.3 and Appendix G (Tables 8–10). Seen-class adaptation results are reported in Section 4.5 and Appendix D (Table 4).

Thank you for the rebuttal.

I appreciate the experiments on autoregressive methods. I think they greatly strengthen the paper. It looks like SPQ is indeed more accurate than these baselines, while also having lower latency. I like the quantification of order dependence and previous-class-prediction dependence.

I still think the novelty of the method is a bit limited. I understand that its application to a multimodal setting is new, as is its application to non-exclusive classification. Normally when I think about the novelty of applying an existing method to a new setting, I consider whether there were any challenges in adapting the method to the new setting, and whether it was surprising or at least non-obvious that the method would work for that setting. I don't see how multimodal multi-label recognition presents any new challenges for the existing method. But I suppose it wasn't obvious a priori that SPQ would be more accurate than AR on multi-label recognition, given that AR gets to see its predictions for previous classes and SPQ doesn't. The latency advantage of SPQ over AR wasn't obvious a priori either. All in all, I think the paper has some nice findings that could be useful to the community.

I will raise my overall rating to a weak accept and my soundness rating to a 3.

AuthorsOpenReview ↗

Appreciation for the Reviewer's Constructive Feedback

We sincerely thank the reviewer for the thoughtful follow-up and encouraging assessment. We appreciate your recognition of the additional autoregressive experiments and their findings on accuracy, efficiency, and prediction dependencies. We also value your perspective on the novelty of the method, which has helped us better clarify the scope and contribution of our work. Thank you again for your constructive feedback and reconsideration of our manuscript.

Comments on the paper

4 comments

We thank the reviewer for the constructive questions concerning the fairness of inference baselines, shared-prefix optimizations, scalability, and computational overhead. We address these concerns with stronger same-backbone comparisons, a revised efficiency figure, and additional implementation-level analyses.

We distinguish the main efficiency comparison from the extended scalability experiments. Section 4.4 (Figure 4) reports inference efficiency on NUS-WIDE with 4–81 candidate classes, while Appendix G provides additional comparisons on 1,000 OpenImages images, including batched independent querying, optimized shared-prefix inference implementations, and scalability up to 400 classes.

1. Fair comparison with cached and batched independent querying

We agree that comparing SPQ only with independent queries that repeatedly process the image would overstate its advantage. The revised Figure 4(a,b) therefore distinguishes Binary QA, which recomputes the image–instruction context for each category, from QA + KV cache, which computes the shared prefix once per image and reuses its key–value states for sequential category-specific suffixes. Both use the same backbone and Yes/No scoring rule as SPQ. We additionally evaluate batched independent querying (BIQ), which batches the suffix queries while retaining the shared prefix cache. For fairness, our chunked-SPQ implementation recomputes the prefix in each chunk, whereas BIQ retains the cached prefix across batches.

Table 1. Additional 400-category efficiency comparison on OpenImages. Latencies are in milliseconds; peak allocated GPU memory is in MiB. The full 4–400-category curves appear in manuscript Figure 4(a,b).

BackboneMethodLatency (ms)Peak memory (MiB)
Qwen2.5-VL-3BSPQ1,497.57,949.5
Qwen2.5-VL-3BBinary QA27,913.77,395.4
Qwen2.5-VL-3BQA + KV cache13,285.67,394.1
Qwen2.5-VL-3BGeneration1,811.08,076.7
InternVL3.5-4BSPQ2,427.39,726.6
InternVL3.5-4BBinary QA23,782.99,209.5
InternVL3.5-4BQA + KV cache14,839.99,218.9
InternVL3.5-4BGeneration3,520.810,704.1

At 400 classes, SPQ is 18.64×/9.80× faster than uncached Binary QA and, more importantly, 8.87×/6.11× faster than prefix-cached QA on Qwen-3B/InternVL-4B, respectively. Relative to candidate-list generation, the speedups are 1.21×/1.45× at this scale. SPQ incurs approximately 555/508 MiB more peak memory than cached QA, making the latency–memory trade-off explicit. Generation can be faster than SPQ at some smaller category counts; moreover, it outputs a positive-label list rather than a complete vector of comparable class scores. This additional 400-category comparison uses 1,000 OpenImages images and three repetitions per configuration. It is separate from the NUS-WIDE efficiency experiment reported in the main Figure 4.

Table 2. Complete chunked-SPQ versus BIQ results on 1,000 OpenImages images. The batch column denotes the tested batch/chunk size; memory is peak allocated GPU memory.

ClassesBatchSPQ (ms)BIQ (ms)SPQ (MiB)BIQ (MiB)
208183.0187.37,3017,349
203278.4117.67,3047,470
2012874.8117.97,3047,470
808650.8483.17,3037,351
8032219.8233.67,3137,574
80128115.9200.67,3347,878
20081,620.61,053.47,3137,361
20032527.7470.77,3247,582
200128259.8436.57,3678,258
40083,208.52,066.37,3327,379
400321,001.1827.87,3437,601
400128502.6717.47,3948,278

Comparing the fastest tested configuration of each method at 20/80/200/400 categories, SPQ requires 74.8/115.9/259.8/502.6 ms versus 117.6/200.6/436.5/717.4 ms for BIQ, giving 1.57×/1.73×/1.68×/1.43× speedups. The benefit is configuration-dependent: BIQ is faster for 80–400 categories at batch size 8 and for 200–400 categories at batch size 32. At 400 categories and batch size 128, SPQ also uses 7,394 MiB compared with 8,278 MiB for BIQ. Thus, SPQ offers an advantage in the best tested latency configurations without claiming to dominate every batching regime.

2. Comparison with optimized shared-prefix inference systems

We additionally evaluate task-adapted Hydragen-style and BatchLLM-style implementations, together with an SGLang pipeline using preprocessing and KV caching. The evaluated systems are adapted to the class-presence workload; we do not claim that these measurements establish a universal ranking of the original serving frameworks. This comparison uses 1,000 OpenImages images, with three timing repetitions per image.

Table 3. Complete shared-prefix system comparison (per-image latency, ms).

ClassesSPQHydragen-styleBatchLLM-styleSGLang
2080.3385.9116.5231.9
8079.6424.8206.1602.7
200110.6543.6353.01,342.9
400180.1603.2596.62,597.7

At 400 classes, the tested SPQ implementation is 3.35×, 3.31×, and 14.42× faster than the Hydragen-style, BatchLLM-style, and SGLang implementations, respectively. These numbers characterize the specified implementations and timing protocol, not every possible server configuration or throughput regime. Although this experiment uses the same OpenImages dataset and sample count as Table 2, its execution and timing configurations differ, so absolute latencies should not be directly compared across experiments.

3. Scaling to 400 categories and computational overhead

To assess large candidate vocabularies, we evaluate unchunked SPQ, constrained autoregressive Yes/No generation (AR), and candidate-guided positive-label generation (CGG) on 1,000 OpenImages images, increasing the number of candidate classes from 4 to 400. SPQ and AR assign a score to every candidate, whereas CGG generates a set of positive category names.

Table 4. Complete 4–400-category scaling comparison on OpenImages. Memory reports the peak allocated GPU memory of SPQ.

ClassesSPQ (ms)AR (ms)CGG (ms)SPQ peak (MiB)
465.2217.6168.47,298
864.4355.5200.17,299
1669.4645.2241.87,302
3280.21,296.8327.67,309
6499.22,508.6630.87,326
81108.72,591.7567.97,333
128143.54,045.4604.37,360
200179.76,610.0804.57,393
400391.913,353.91,403.07,506

At 400 categories, SPQ is 34.1× faster than AR and 3.58× faster than CGG. From 4 to 400 classes, SPQ latency increases by approximately 6.0×, versus 61.4× for AR, while SPQ peak allocated memory increases only 208 MiB (7,298 to 7,506 MiB). This supports scalability in the tested inference configuration, although SPQ's runtime still grows with its packed sequence length. The absolute times in response Tables 1–4 should not be conflated: they come from different processing, batching, or timing configurations even where dataset and sample count match.

We also report the actual packed input lengths and the number of permitted query–key attention connections. For a shared prefix of length PP and category suffix lengths LiL_i, the class-isolated attention pattern permits P(P+1)/2+P∑iLi+∑iLi(Li+1)/2P(P+1)/2 + P\sum_i L_i + \sum_i L_i(L_i+1)/2 connections. In the reported OpenImages setting, the shared prefix is 104 tokens, including 64 visual tokens.

Table 5. Complete packed-token and permitted-attention counts.

ClassesSuffix tokensTotal tokensPermitted connections
2011521917,850
8043854252,539
2001,1491,253129,198
4002,3542,458259,055

This accounting demonstrates that the shared prefix is represented once and that cross-suffix attention connections are excluded. Crucially, permitted connections are not measured FLOPs: dense GPU attention kernels may still compute masked positions. Our latency and peak-memory measurements, rather than a theoretical sparse-mask count alone, support the practical efficiency claim.

4. Generation-length sensitivity, answer verbalizers, and the zero-shot setting

We test whether CGG performance is an artifact of its generation-length limit. Candidate order and evaluation samples are held fixed while varying the maximum output length.

Table 6. Complete CGG generation-length sensitivity results. Precision, recall, and Micro-F1 are percentages.

DatasetMaximum new tokensPrecisionRecallMicro-F1Average tokensOutputs at limit (%)
VOC16–12853.9199.2069.852.340.00
COCO1673.8587.2579.993.290.77
COCO3271.3087.3778.523.410.44
COCO6471.1387.4178.433.420.00
COCO12871.1387.4178.433.420.00

VOC results are unchanged across the tested limits; on COCO, at most 0.77% of outputs reach the limit, and increasing the limit does not consistently improve Micro-F1. Frequent output truncation therefore does not explain CGG's substantial sensitivity to candidate order. We also examine five Yes/No alternative verbalizer pairs in Section 4.5 and Figure 4(c): Yes/No leads in four of eight backbone–dataset settings, while True/False and positive/negative lead in two each. We retain a fixed Yes/No verbalizer for the main comparisons rather than selecting the best answer pair using test results. Finally, zero-shot means that recognition is evaluated on benchmark-defined unseen categories without task-specific training; it does not imply that these concepts were absent from MLLM pretraining.

Section 4.4 and Figure 4(a,b) report the main NUS-WIDE efficiency comparison (4–81 classes), with measurement details in Appendix A.3. An additional 400-class OpenImages comparison is provided in response Table 1. Appendix G.1–G.4 (Tables 8–11) presents batched independent querying, optimized shared-prefix inference, scalability up to 400 classes, and computational overhead. Appendix F.3 (Table 7) reports generation-length sensitivity, while Section 4.5 and Figure 4(c) examine answer-verbalizer sensitivity.

Area_Chair_r2NNOpenReview ↗

Please check authors' rebuttal and participate in the discussion

The discussion phase is coming to an end. Please read the authors’ rebuttal and post a brief follow-up in the discussion thread. Let us know which of your concerns the response addresses and which remain open.

Please post any follow-up questions in the thread visible to the author as soon as possible, so that the author has time to respond and provide clarification. I also encourage you to engage with one another’s assessments—especially where your perspectives differ—and to note whether any counterarguments or discussions lead you to revise your own assessment.

Thank you for your effort and thoughtful review. Thank you again!

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.