- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
To address the limitation of existing distillation methods that directly adopt teacher predictions as supervision, the proposed method seeks to identify discriminative information and assign tailored distillation signals across different classes and samples. The reported results demonstrate advantages over methods that do not leverage foundation models.
Critical strengths
This paper investigates how different modalities contribute to individual predictions and which teacher signals ought to be transferred to the student model. This work is beneficial for distillation research that applies foundation models to modality-fusion tasks. The experimental results demonstrate the advantages brought by the foundation model and the designed distillation scheme.
Critical weaknesses
- Only one foundation model is adopted in the experiments, while the other competing methods do not rely on foundation models. The authors are suggested to evaluate the proposed distillation scheme on multiple foundation models to better demonstrate its effectiveness.
- Compared with other distillation methods should be included.
- Comparisons of model size, training speed and inference speed should be supplemented.
- Can the proposed distillation scheme be extended to other modality-fusion tasks?
- The manuscript requires further revision. In particular, several critical technical issues are not clearly described
Questions for authors
Compared with so many distillation methods, what are the main distributions of the paper? The theoretic discussion and experimental comparison should be provided.