- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates rare-class suppression during vision-language model (VLM) finetuning on class-imbalanced datasets. The authors identify a counterintuitive phenomenon termed the matched-prompt penalty. Based on this observation, they propose cross-prompt inference, a simple inference-time strategy that requires no additional model training. The paper evaluates this phenomenon using six datasets and six VLMs, with controlled train–evaluation prompt matrices. The results demonstrate consistent improvements in rare-class recall, although sometimes at the expense of frequent-class performance.
Critical strengths
-
Interesting and non-obvious empirical finding with convincing controlled evidence.
-
Comprehensive experimental analysis and useful mechanistic investigation.
-
The TuCo analysis is particularly valuable: the intervention that progressively removes finetuning-induced computation substantially reduces the matched-prompt penalty, providing evidence consistent with the proposed explanation.
Critical weaknesses
-
Although the experiments cover six datasets, each is reduced to only 20 selected classes, consisting of 10 frequent and 10 rare classes. The class selections and imbalance configurations are carefully constructed, often emphasizing visually confusable categories.
-
Although cross-prompt inference consistently improves rare-class recall, these improvements can come at meaningful costs to frequent-class and overall performance. For example, Table 1 shows that average mismatched inference reduces overall recall on CUB-200, Stanford Dogs, and PlantNet-300K under imbalanced finetuning.
-
The paper lacks direct comparisons with established inference-time class-prior corrections or logit calibration methods. Consequently, it remains unclear whether cross-prompt inference offers a competitive improvement in overall classification quality relative to simpler, potentially cheaper alternatives.
Questions for authors
- How does cross-prompt inference compare with standard inference-time bias correction methods, such as logit adjustment or class-prior calibration, under comparable validation-data and computational budgets?
I will adjust my score if authors can address my questions.