acceptodds
Withdrawn from ICLR 2027

Cross-Prompt Inference Mitigates Rare-Class Suppression in Imbalanced VLM Finetuning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

When a vision-language model is finetuned on class-imbalanced data, rare-class recall can fall well below the model’s own zero-shot performance. In this regime, we uncover a surprising matched-prompt penalty whereby the standard choice of reusing the descriptive finetuning prompt at inference can yield lower rare-class recall than using a different descriptive prompt. We call this inference-time strategy cross-prompt inference. We evaluate diverse VLMs across six datasets and find that cross-prompt inference consistently recovers rare-class recall, with varying trade-offs in frequent-class recall. The effect reflects a train-eval prompt interaction rather than prompt quality alone, and does not arise with non-descriptive prompt changes. It is strongest when imbalance and visual confusability produce substantial rare-class suppression, and is weak or absent when such suppression does not occur. Using Tuning Contribution (TuCo), which decomposes a model’s computation into pretrained and finetuning parts, we find that cross-prompt inference reduces reliance on finetuning-induced computation and shifts the model toward its pretrained component. Reducing that finetuning contribution also substantially reduces the cross-prompt advantage. These results show that rare-class knowledge suppressed by imbalanced finetuning can remain accessible under a different inference prompt.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-language models, Class-imbalanced learning, Long-tailed recognition, Fine-tuning, Prompt sensitivity

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YtQPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates rare-class suppression during vision-language model (VLM) finetuning on class-imbalanced datasets. The authors identify a counterintuitive phenomenon termed the matched-prompt penalty. Based on this observation, they propose cross-prompt inference, a simple inference-time strategy that requires no additional model training. The paper evaluates this phenomenon using six datasets and six VLMs, with controlled train–evaluation prompt matrices. The results demonstrate consistent improvements in rare-class recall, although sometimes at the expense of frequent-class performance.

Critical strengths
  1. Interesting and non-obvious empirical finding with convincing controlled evidence.

  2. Comprehensive experimental analysis and useful mechanistic investigation.

  3. The TuCo analysis is particularly valuable: the intervention that progressively removes finetuning-induced computation substantially reduces the matched-prompt penalty, providing evidence consistent with the proposed explanation.

Critical weaknesses
  1. Although the experiments cover six datasets, each is reduced to only 20 selected classes, consisting of 10 frequent and 10 rare classes. The class selections and imbalance configurations are carefully constructed, often emphasizing visually confusable categories.

  2. Although cross-prompt inference consistently improves rare-class recall, these improvements can come at meaningful costs to frequent-class and overall performance. For example, Table 1 shows that average mismatched inference reduces overall recall on CUB-200, Stanford Dogs, and PlantNet-300K under imbalanced finetuning.

  3. The paper lacks direct comparisons with established inference-time class-prior corrections or logit calibration methods. Consequently, it remains unclear whether cross-prompt inference offers a competitive improvement in overall classification quality relative to simpler, potentially cheaper alternatives.

Questions for authors
  1. How does cross-prompt inference compare with standard inference-time bias correction methods, such as logit adjustment or class-prior calibration, under comparable validation-data and computational budgets?

I will adjust my score if authors can address my questions.

Reviewer_3aLTOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies rare-class suppression when generative VLM classifiers are finetuned on imbalanced datasets. It finds that switching from the descriptive training prompt to another descriptive evaluation prompt often improves rare-class recall, sometimes at a cost to frequent classes. Experiments span six datasets and six VLMs, with imbalance, similarity, oversampling, LoRA and non-descriptive-prompt controls. A TuCo intervention studies the role of decoder-layer adaptation. Prompt ensembling and validation-based prompt selection provide practical variants.

Critical strengths

S1. The matched-prompt penalty is an interesting empirical finding. The train/evaluation prompt matrices expose a nontrivial interaction between descriptive prompting and imbalanced finetuning.

S2. The authors acknowledge the 20-class scope and frequent-class trade-off. Appendix E goes beyond test-selected best prompts by evaluating ensembling and small-validation-set selection.

Critical weaknesses

W1. The practical gain needs comparison at matched frequent-class cost. Oversampling and prompt ensembling are evaluated, but no trade-off comparison with score-level prior correction or pretrained/finetuned interpolation shows whether the prompt switch provides a better operating point.

W2. The mechanism is not identified by the current lambda intervention. Table 5 reports lower TuCo under mismatch, but the non-descriptive mismatches in Table 2 lack the corresponding TuCo control.

W3. The magnitude and scope of practical recovery need clearer qualification. The main claims should distinguish test-selected upper bounds, validation-selected performance and label-free ensemble performance, including their uncertainty and resource costs.

W4. Eq. (3) defines best mismatch using a maximum over evaluation-prompt rare recall, and Figure 4 reports best-mismatch curves. These are useful analytical upper bounds, not a deployment selection rule. Appendix E already evaluates 200 validation-selection splits and an ensemble that does not require selection labels. The main claims should distinguish average mismatch, test-selected best mismatch, ensemble performance and validation-selected performance.

W5. Section 4.3's claim that averaged cross-prompt inference exceeds zero-shot overall recall on every dataset needs qualification. Table 1 reports 84.8 for Food-101 under imbalanced training versus zero-shot 86.4; the oversampled average is 86.5. Please identify the training regime intended by the statement. The matched imbalanced model's 81.9 also makes the pretrained reference important when describing practical improvement.

Questions for authors

The prompt-mismatch finding is interesting and supported by a substantial controlled study. However, the current evidence does not identify the proposed mechanism, and the practical rare-class recovery is not established relative to simple alternatives at comparable frequent-class cost. A mechanism-discriminating control or a convincing matched-cost advantage could move my recommendation toward acceptance.

Reviewer_6C42OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work addresses a specific issue in the context of Vision-Language Models (VLMs): when training data contains a significant imbalance (i.e., with some classes having many samples and others very few), fine-tuning can cause the model to become biased toward the more frequent classes, leading to a decline in rare-class recall. The authors note that, in such scenarios, the descriptive prompts used during fine-tuning may not be the most effective ones for inference. They propose a solution that simply switching to a different descriptive prompt for classification after the fine-tuning process is complete to preserve the rare-class recall.

Critical strengths

None

Critical weaknesses

The novelty of the core contribution is insufficient. According to the claims made in the introduction, the authors argue that "cross-prompt inference shifts computation away from imbalanced fine-tuning and toward the pre-trained component, recovering rare-class knowledge suppressed by fine-tuning." However, the related work section reveals that several existing studies have already put forward the same perspective—namely, that "pre-trained capabilities can be recovered by shifting prompts away from the fine-tuning distribution." This paper merely applies this existing concept to the specific context of VLM classification under class imbalance. In this way, it offers no new underlying principles or generalizable conclusions that extend beyond what is already known by the community.

Even assuming the primary focus of this work is to "verify and analyze existing phenomena within downstream application scenarios," the current exploration is insufficient to establish that the work meets the necessary standards. The experiments provided do not adequately disentangle the respective contributions of descriptive content, text length, semantic shifts, and the degree of deviation from the training prompts. Crucial questions—such as why descriptive prompts used during training trigger stronger suppression—remain unaddressed and unverified. Furthermore, the study lacks direct comparisons with other baselines designed to tackle the long-tailed recognition, as well as any analysis regarding the relative merits of the underlying mechanisms.

Questions for authors

The same as mentioned in weaknesses.

  1. What is the conceptual novelty of cross-prompt inference beyond prior findings on recovering pretrained capabilities through prompt shifts?
  2. Can the authors provide stronger evidence for the proposed method and its advantages over alternative approaches to class imbalance? Specifically, how can one decouple factors such as descriptive content, prompt length, semantic shifts, and the deviation from the prompts used during training, in order to explain why matching descriptive prompts trigger a more pronounced rare-class recall suppression? ​​Furthermore, how does this cross-prompt inference approach perform in comparison to existing baselines for long-tailed learning and class imbalance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.