acceptodds
Under review as a conference paper at ICLR 2027

DualDistill: Transferring Generative Recognition into Efficient Discriminative Prediction for Annotation-Free Multi-Label Visual Recognition

Abstract

Traditional multi-label visual recognition relies heavily on human-annotated data, making it costly to scale to large datasets and label vocabularies. Multimodal Large Language Models (MLLMs) provide a promising alternative by performing annotation-free recognition through autoregressive label generation. However, this generative paradigm incurs substantial inference overhead due to task-prompt encoding and token-by-token decoding. In this paper, we investigate how to transfer the generative recognition capability of MLLMs into flexible and efficient discriminative prediction. We identify two key challenges in this transition: removing the task-aware prompt discards valuable candidate-label semantics, while directly optimizing MLLMs with a discriminative classification objective introduces a mismatch with their original generative training paradigm. To address these challenges, we propose DualDistill, a progressive generation-to-discrimination framework consisting of Cross-Paradigm Distillation (CPD) and Cross-Token Distillation (CTD). CPD first converts generative recognition into prompt-aware discriminative prediction while preserving the generative learning signal. CTD then transfers the resulting prompt-aware capability to an image-only pathway for efficient inference. Extensive experiments on multiple benchmarks demonstrate that DualDistill substantially improves both recognition performance and inference efficiency over strong MLLM fine-tuning and annotation-free baselines, achieving up to a 14 times inference speedup while maintaining strong classification performance without requiring any human annotations.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multi-label visual recognition, multimodal large language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_mjT8OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes DualDistill, a progressive generation-to-discrimination framework for annotation-free multi-label visual recognition. The method aims to transfer the recognition capability of multimodal large language models into efficient discriminative prediction. It introduces Cross-Paradigm Distillation (CPD), which combines generative-to-discriminative supervision with generative learning preservation, and Cross-Token Distillation (CTD), which transfers prompt-aware knowledge from a prompt-side classification token to an image-side token. The resulting image-side classifier performs multi-label prediction without task-prompt encoding or autoregressive decoding. Experiments on three benchmark datasets demonstrate strong recognition performance and substantial inference speedups compared with generative recognition and existing annotation-free approaches.

Critical strengths
  1. The paper addresses an important limitation of MLLM-based multi-label recognition, namely the high inference cost of autoregressive generation. The progressive transfer from prompt-aware to image-only discrimination provides a clear and practically meaningful solution.

  2. DualDistill achieves substantial improvements over existing annotation-free methods while providing significant inference speedups. The evaluation across different datasets and MLLM backbones, together with ablation studies and visualizations, provides encouraging evidence of the framework's effectiveness.

Critical weaknesses
  1. The necessity of the progressive distillation design is not fully established. Although CPD and CTD are individually motivated, their underlying operations largely rely on established techniques, including pseudo-label supervision, generative auxiliary training, and prediction-level knowledge distillation. The current experiments do not sufficiently demonstrate why the proposed progressive structure is preferable to simpler distillation alternatives.

  2. The framework relies entirely on labels generated by a frozen teacher MLLM, but the influence of pseudo-label errors, especially missing positive labels, is not sufficiently investigated. It remains unclear how robust the proposed approach is to different levels of teacher quality.

  3. The reported inference speedups are promising, but much of the efficiency advantage comes from replacing autoregressive generation with direct classification. Stronger comparisons with simpler discriminative adaptations would help isolate the contribution of DualDistill itself. Furthermore, the total cost of teacher annotation generation and student training is not fully discussed.

Questions for authors
  1. (Most likely to change my score) Could the authors provide a controlled comparison with simpler knowledge distillation approaches, such as directly distilling the teacher's predictions into an image-only classification token or using a conventional pseudo-label-trained discriminative head? This would help clarify whether the progressive CPD–CTD architecture provides benefits beyond standard discriminative adaptation and knowledge distillation.

  2. How sensitive is DualDistill to the quality of teacher-generated annotations? In particular, how does the model handle missing positive labels or incorrect categories in the teacher responses, and would the performance remain stable with a weaker teacher?

  3. In Table 3, introducing G2G does not consistently improve the image-side classifier, particularly on Objects365. Could the authors explain why preserving the generative objective benefits the prompt-side pathway more consistently than the image-side pathway, and whether G2G remains necessary when CTD is applied?

  4. The proposed CTD transfers predictions from the prompt-side token to the image-side token using KL divergence. Could the authors clarify why this prediction-level distillation is preferable to feature-level distillation or other alignment strategies? Would the two approaches provide complementary benefits?

Reviewer_sCHAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes DualDistill, a framework for transferring generative recognition from an MLLM to discriminative multi-label prediction. The method introduces a prompt-side classification token and combines two delicate objectives. It then applies Cross-Token Distillation to transfer the prompt-aware prediction to an image-only token. Experiments show substantial throughput improvements and competitive performance.

Critical strengths

[+]. Great writing, easy-to-follow.

[+]: Addresses an important practical problem, i.e., reducing the inference cost of MLLM-based annotation-free multi-label recognition.

[+]: The combination of G2D, G2G, and CTD is conceptually coherent.

[+]: Experiments are extensive, covering cover multiple datasets and backbones.

Critical weaknesses

[-]: I am concerned that, injecting deterministic classification capabilities into MLLMs may compromise their general VQA and reasoning abilities. If this is indeed the case, the applicability of DualDistill would be limited. In other words, the authors must either show that MLLMs can retain both deterministic and general capabilities under the DualDistill, or, justify that such classification-specialized training holds unique application prospects. Noticing the recent Jev-like models, if models trained via DualDistill could evolve into a more distinct 'System 1' model such as Jev, the significance of DualDistill would be enhanced.

[-]: The methodological novelty seems a combination of existing methods. The conceptual advance is moderate rather than fundamental.

[-]: The evidence for the claimed optimization mismatch and semantic transfer is incomplete. Attention visualizations and hidden-state similarity do not establish causal improvement.

[-]: Comparisons may not be fully fair. The authors should report data splits, prompt construction, pseudo-label parsing, implementation details, and statistical variation. Training cost for process likr teacher annotation generation, memory usage, latency, and batch-size sensitivity are not reported.

Questions for authors

See weaknesses above.

Reviewer_nRkxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to transfer the generative recognition ability of MLLMs into discriminative prediction for annotation-free multi-label recognition. Autoregressive generation in existing MLLMs causes high inference costs for multi-label recognition. To address this issue, the authors utilize an existing MLLM as a teacher model and aim to preserve the original generative ability of MLLMs while improving discriminative prediction through Cross-Paradigm Distillation and Cross-Token Distillation. Finally, they achieves strong performance compared to annotation-free multi-label recognition methods.

Critical strengths
  1. Well-designed learning strategy for enhancing the discriminative prediction ability and removing the limitation of auto-regressive generation.

  2. They demonstrated strong performance of multi-label recognition without annotations.

Critical weaknesses
  1. The existing MLLMs is used as teacher models. They are not perfect and can generate false predictions. Therefore, student model can learn the false prediction and be slightly biased to error output of teacher model. The analysis of imperfect teacher model's output and dependency of student on error output is required.

  2. Limited flexibility of the proposed method. While the method is based on the annotation free, training is required and it can not predict other classes. The existing method such as TagCLIP is training-free and they can be easily extended to other classes and domains.

Questions for authors

My main questions concern the limitations related to imperfect teacher predictions and the flexibility of the proposed method. How sensitive is DualDistill to incorrect pseudo-labels generated by the teacher MLLM? Can the proposed method handle new categories without additional training, or is retraining required when the category vocabulary changes?

1 comment

Thank you for the rebuttal. Most of my concern is resolved and I will raise my score to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.