acceptodds
Under review as a conference paper at ICLR 2027

Rarity-Aware Vision-Language Benchmarking and Prompting

Abstract

Prompt learning for vision-language models is predominantly evaluated under balanced supervision, leaving the effects of class imbalance on recognition and transfer insufficiently understood. We introduce TailPromptBench, a benchmark for long-tailed prompt generalization that varies class imbalance under a fixed annotation budget and evaluates recognition across frequency groups, base-to-new transfer, and robustness to visual domain shift. This design separates unequal supervision from changes in training-set size. We further propose RAP, a rarity-aware training parameterization for prompt learning based on a shared residual context. Its contribution increases with class rarity, providing stronger auxiliary corrections for underrepresented classes without introducing class-specific parameters. Joint optimization allows rarity-dependent contexts to shape the prompt parameters retained for deployment. The residual is discarded at inference, preserving the original structure and cost without requiring frequency estimates for unseen categories. Experiments show stronger tail-class recognition and average base-to-new harmonic-mean gains of up to 2.8 percentage points, with benefits extending to shifted domains. Ablations support the importance of aligning residual adaptation with the actual class-frequency structure.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
vision-language models, prompt learning, long-tailed learning, base-to-new generalization, domain generalization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bem3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies prompt learning for vision-language models under class-imbalanced supervision. The authors introduce TailPromptBench, which evaluates long-tailed recognition while keeping the total number of training samples fixed. They also propose RAP, which adds a shared residual to the prompt during training, with its contribution determined by class frequency. The residual is removed during inference.

Critical strengths
  1. I appreciate the fixed-budget evaluation protocol. Keeping the total number of training samples unchanged makes it easier to study the effect of class-imbalance without introducing differences in training-set size. The evaluation of both tail-class recognition and base-to-new generalization is also useful.

  2. The proposed method is simple and introduces little additional training overhead. The results show consistent improvements over CoOp and CoCoOp under severe imbalance. The experiments with shuffled and constant rarity coefficients also provide some evidence that class-frequency information contributes to these improvements.

Critical weaknesses
  1. My main concern is the lack of comparisons with existing methods specifically developed for long-tailed VL learning. The main experiments only compare RAP with CoOp and CoCoOp, neither of which was designed to handle class imbalance. The paper discusses more recent methods, including Candle and LAMP, but does not provide experimental comparisons with them. As a result, the current experiments show that rarity-aware prompting improves standard prompt learning, but they do not establish its advantage over existing imbalance-aware approaches.

  2. The methodological novelty is somewhat limited. The main idea is to scale a shared residual prompt according to class frequency, which is a relatively straightforward modification. I am not convinced that this is substantially different from introducing frequency-dependent adjustments during optimization. Comparisons with standard class reweighting or logit adjustment would help clarify the contribution.

  3. The motivation for removing the rarity-dependent residual at inference is understandable, since class frequencies are unavailable for unseen categories. However, the paper does not sufficiently investigate why discarding this residual leads to better generalization. I would also like to see a more detailed analysis of the trade-off between head- and tail-class accuracy, since Figure 5 shows that tail-class improvements often come with noticeable decreases in head-class performance.

Questions for authors
  1. Can the authors compare RAP with recent LT VL adaptation methods, such as Candle and LAMP, under the same TailPromptBench protocol? This is particularly important for assessing the contribution of the proposed method.

  2. How does RAP compare with simpler imbalance-handling techniques, such as class-balanced loss, class reweighting, or logit adjustment?

  3. Have the authors evaluated alternatives to completely removing the residual at inference? For example, what happens if a fixed residual coefficient is used for all test classes?

2 comments
AuthorsOpenReview ↗

Response to Reviewer bem3

We thank the reviewer for the actionable suggestions on long-tailed baselines and inference-time residuals.

W1 & Q1. Comparisons with long-tailed VLM methods

A1. We have added direct comparisons with Candle and LAMP on the same TailPromptBench splits and fixed training-image budget. The table reports six-dataset averages at ρ=64\rho=64, with H averaged over dataset-level harmonic means. We also include class-balanced loss (CB, β=0.99\beta=0.99) and logit adjustment (LA, τ=1.0\tau=1.0), applied to both prompt learners and combined with RAP.

MethodBaseNewH
CoOp76.6769.3372.46
+ CB77.9470.7574.17
+ LA78.4169.9673.94
+ RAP79.5271.6975.18
+ RAP + CB80.4571.1075.49
+ RAP + LA80.9969.4874.79
CoCoOp74.3670.1371.93
+ CB77.3969.7373.04
+ LA78.3868.9572.88
+ RAP79.0371.4274.71
+ RAP + CB80.6875.7578.01
+ RAP + LA80.9774.1477.17
Candle77.2271.3173.67
LAMP78.1369.9373.81

CoOp+RAP and CoCoOp+RAP achieve mean H of 75.18 and 74.71, respectively, compared with 73.67 for Candle and 73.81 for LAMP. We have added these comparisons and their experimental settings to Section 5.

W2 & Q2. Difference from class reweighting and logit adjustment

A2. Class-balanced loss changes the weight of a training example; logit adjustment changes the class scores. RAP changes the text representation used to compute those scores. Each candidate class receives the context Vθ(x)+rcΔVV_\theta(x)+r_c\Delta V before the frozen text encoder. Rarity therefore affects the representations of both the positive class and the negative classes in the softmax. The original prompt and shared residual are trained jointly; inference retains the learned original prompt. The table above compares these choices directly: RAP has higher H than either CB or LA on both learners, and CB provides a further gain when combined with RAP. We have made the distinction from loss weighting and logit correction explicit in Section 4.

W3 & Q3. Alternatives to removing the residual

A3. We have not evaluated a non-zero coefficient shared by all test classes, Vθ⋆(x)+γΔV⋆V_{\theta^\star}(x)+\gamma\Delta V^\star. Our available experiment varies the retained training residual, Vθ⋆(x)+αrcΔV⋆V_{\theta^\star}(x)+\alpha r_c\Delta V^\star, using the same trained checkpoints on base classes. Here α\alpha is shared but rcr_c still varies by class. The table averages the six datasets at ρ=64\rho=64:

Methodα\alphaBaseHeadMidTailGap (pp) ↓
CoOp0.0079.5185.7480.9668.9716.77
0.2579.3287.4281.1269.2018.23
0.5079.0588.8981.4965.4523.45
0.7578.4990.1080.7561.2128.89
1.0076.3690.5978.9255.7934.80
CoCoOp0.0079.0485.9379.5967.5218.41
0.2578.6187.4878.9165.4422.04
0.5077.7488.5877.8162.9225.66
0.7576.3589.3176.2059.7129.60
1.0074.4589.7273.8955.9733.75

Removal gives the highest base accuracy in this sweep. Full retention raises head accuracy but reduces base accuracy from 79.51 to 76.36 for CoOp and from 79.04 to 74.45 for CoCoOp; tail accuracy falls by 13.18 and 11.55 points. After removal, inference uses the jointly trained original prompt, including CoCoOp's conditioning network. We have added the residual-retention sweep and its precise inference rule to Section 5.

W3. Analysis of the trade-off between head- and tail-class accuracy

A3 (continued). For CoOp, the default is not the best head-tail choice in our sweep: λΔ=1\lambda_\Delta=1 improves both head and tail accuracy over λΔ=0.1\lambda_\Delta=0.1. The latter was selected for base-to-new H. For CoCoOp, the default λΔ=0.001\lambda_\Delta=0.001 lies on the observed Pareto frontier; the alternatives that improve head accuracy lose tail accuracy. We have added the head-tail Pareto analysis in Appendix G and discussed it in Section 5.

Reviewer_bem3OpenReview ↗

Comment

Some of my concerns have been addressed, so I will raise my score to 3.

Reviewer_xMAbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces TailPromptBench, which redistributes a fixed annotation budget to study long-tailed prompt generalization. RAP adds a shared, rarity-scaled context residual during training and removes it for inference. On CLIP ViT-B/16 with CoOp and CoCoOp, it improves the six-dataset mean base-to-new harmonic mean by 2.72 and 2.79 points at severe imbalance, with gains in tail recognition and shifted-domain evaluation.

Critical strengths
  1. Holding the annotation budget fixed can help separate supervision imbalance from changes in training-set size.
  2. RAP preserves the underlying learner's inference architecture and requires no frequency estimates for unseen classes. Its additional training parameters are independent of vocabulary size.
  3. The constant and shuffled-rarity controls provide evidence that the frequency assignment matters. The paper also reports the head-class costs and exceptions to new-class and mild-imbalance improvements.
Critical weaknesses
  1. The experiments compare RAP with CoOp, CoCoOp, and its own variants, although Section 2 discusses methods designed for long-tailed adaptation. Frequency-based loss reweighting or logit adjustment on the same learners would be particularly informative. Comparisons with compatible long-tailed methods under the fixed-budget protocol are needed to establish RAP's contribution beyond improving the unmodified learners.

  2. Eq. (5) correctly shows that rarity changes optimization, and the context-norm bound limits token displacement. Neither establishes why the retained prompt should benefit after the auxiliary correction is removed. Table 3 supports the frequency assignment but does not isolate this step. On base classes, where frequencies are known, evaluating the same trained model with and without the residual would help reveal the deployment gap and which parameters carry the tail gains.

  3. Results average three seeds, but no standard deviations or paired-seed differences are reported. This matters for the 0.74-point EuroSAT/CoCoOp and 0.80-point UCF101/CoOp gains in Table 1, and for selecting among nearby settings in Table 8. The positive mean results do not establish that these individual differences are stable.

Questions for authors

See weaknesses

1 comment
AuthorsOpenReview ↗

Response to Reviewer xMAb

We thank the reviewer for the detailed comments, particularly the suggested residual-retention and paired-seed comparisons.

W1. Comparisons with long-tailed learning methods

A1. We added class-balanced loss (CB) and logit adjustment (LA) on the same fixed-budget splits, together with Candle. CB uses β=0.99\beta=0.99, LA uses τ=1.0\tau=1.0, and Candle retains its original optimization settings. The table reports six-dataset averages at ρ=64\rho=64, with H averaged over dataset-level harmonic means.

MethodBaseNewH
CoOp76.6769.3372.46
+ CB77.9470.7574.17
+ LA78.4169.9673.94
+ RAP79.5271.6975.18
+ RAP + CB80.4571.1075.49
+ RAP + LA80.9969.4874.79
CoCoOp74.3670.1371.93
+ CB77.3969.7373.04
+ LA78.3868.9572.88
+ RAP79.0371.4274.71
+ RAP + CB80.6875.7578.01
+ RAP + LA80.9774.1477.17
Candle77.2271.3173.67

RAP has higher H than CB and LA on both learners, and higher H than Candle. Adding CB to RAP gives a further gain on both learners, reaching 75.49 for CoOp and 78.01 for CoCoOp. We have added these comparisons and their settings to Section 5.

W2. Motivation for removing the rarity-dependent residual

A2. We evaluated the same trained model with and without its residual, as suggested. On base classes, we vary only α\alpha in Vθ⋆(x)+αrcΔV⋆V_{\theta^\star}(x)+\alpha r_c\Delta V^\star; the learned prompt and residual stay fixed. The results average the six datasets at ρ=64\rho=64. The base and tail gains remain after the explicit rarity correction is removed. During training, the gradients for the retained prompt are evaluated at rarity-conditioned contexts (Eq. 5), so removing ΔV\Delta V does not return the prompt to its baseline-trained parameters. CoCoOp retains both the learned context and its image-conditioning network.

Methodα\alphaBaseHeadMidTailGap (pp) ↓
CoOp0.0079.5185.7480.9668.9716.77
0.2579.3287.4281.1269.2018.23
0.5079.0588.8981.4965.4523.45
0.7578.4990.1080.7561.2128.89
1.0076.3690.5978.9255.7934.80
CoCoOp0.0079.0485.9379.5967.5218.41
0.2578.6187.4878.9165.4422.04
0.5077.7488.5877.8162.9225.66
0.7576.3589.3176.2059.7129.60
1.0074.4589.7273.8955.9733.75

Removing the residual raises base accuracy by 3.15 points for CoOp and 4.59 for CoCoOp compared with full retention, and tail accuracy by 13.18 and 11.55 points. It gives the highest base accuracy and smallest head-tail gap in this sweep. We have added this same-model comparison to Section 5 and explained the retained parameters in Section 4.

W3. Statistical variability across seeds

A3. The UCF101/CoOp gain is positive in all three seeds: +1.54, +0.10, and +0.64 points, giving a paired mean and sample standard deviation of +0.76±0.73+0.76\pm0.73. The small EuroSAT/CoCoOp gain is seed-dependent: +0.14±8.63+0.14\pm8.63, with an improvement in only one of the three runs.

Across the six datasets and two learners, nine of the twelve comparisons improve in every seed. EuroSAT/CoOp and UCF101/CoCoOp also have seed-dependent gains. The 0.74-point EuroSAT difference in Table 1 belongs to CoCoOp.

Table 1 computes each dataset's H from seed-averaged base and new accuracies, then averages H across datasets. The paired analysis computes H within each seed before taking the RAP-minus-baseline difference. The different aggregation order explains why the paired means need not equal the aggregate differences in Table 1.

In the regularization sweep, the defaults still have the highest observed mean H across seeds: 75.142±1.31575.142\pm1.315 for CoOp at λΔ=0.1\lambda_\Delta=0.1, and 74.527±0.70074.527\pm0.700 for CoCoOp at λΔ=0.001\lambda_\Delta=0.001. Nearby choices give 74.935±1.33174.935\pm1.331 for CoOp at λΔ=1\lambda_\Delta=1, and 74.169±0.43574.169\pm0.435 for CoCoOp at λΔ=0.3\lambda_\Delta=0.3. These three-seed results do not establish a reliable ordering between the nearby coefficients. The defaults were selected using the six-dataset benchmark mean, without a separate validation set. We have added the paired-seed results in Appendix E, the sweep variability in Appendix G, and the aggregation definitions in Section 5.

Reviewer_EcQ9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies prompt learning for VLMs under long-tailed supervision. It introduces TailPromptBench, which evaluates VLM under multiple shift settings. It also proposes RAP, a parameterization method that adds a residual context scaled by class rarity. Experiments with CoOp and CoCoOp show consistent improvements under severe imbalance; especially for tail-class recognition and base-to-new.

Critical strengths

[+]: Good writing.

[+]: Provided a benchmark for long-tailed prompt generalization

[+]: RAP has a clear and technically simple formulation, i.e., a shared residual is modulated by class rarity during training and removed at inference.

[+]: The experiments are extensive and the main improvements are consistent. The paper reports important trade-offs, including head-class degradation and occasional decreases under mild imbalance.

Critical weaknesses

[-]: The method is compared mainly against CoOp and CoCoOp, without direct comparisons to class-balanced sampling, loss reweighting, logit adjustment, focal/class-balanced losses, or existing long-tailed VLM methods.

[-]: The long-tailed distributions are constructed by randomly assigning class identities to frequency ranks, so the results may not reflect real long-tailed data where frequency correlates with semantic difficulty and visual diversity.

[-]: Experiments use only CLIP ViT-B/16 and two prompt learners, leaving the generality of RAP under boarder settings unclear.

[-]: The theoretical explanation shows how the residual affects optimization, but does not establish why removing the residual after training should preserve or improve performance.

Questions for authors

See weaknesses above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer EcQ9 (Part 1/2)

We thank the reviewer for the careful reading and actionable suggestions.

W1. Comparisons with long-tailed learning methods

A1. RAP has higher mean H than class-balanced loss (CB), logit adjustment (LA), and Candle under the same TailPromptBench splits. We added CB with β=0.99\beta=0.99 and LA with τ=1.0\tau=1.0 to both CoOp and CoCoOp, and evaluated Candle using its original optimization settings. The table reports six-dataset averages at ρ=64\rho=64; H is the average of the dataset-level harmonic means.

MethodBaseNewH
CoOp76.6769.3372.46
+ CB77.9470.7574.17
+ LA78.4169.9673.94
+ RAP79.5271.6975.18
+ RAP + CB80.4571.1075.49
+ RAP + LA80.9969.4874.79
CoCoOp74.3670.1371.93
+ CB77.3969.7373.04
+ LA78.3868.9572.88
+ RAP79.0371.4274.71
+ RAP + CB80.6875.7578.01
+ RAP + LA80.9774.1477.17
Candle77.2271.3173.67

Class balancing also helps when combined with RAP: H rises from 75.18 to 75.49 for CoOp and from 74.71 to 78.01 for CoCoOp. We have added the baseline comparisons and their settings to Section 5.

W2. Random assignment of class frequencies

A2. Random assignment omits correlations between class frequency and visual difficulty. We tested two departures from that protocol on ImageNet and Food101: assigning frequency ranks using frozen-CLIP recognition difficulty, or using within-class visual-feature dispersion. The class-count profile and total training budget stay fixed; only the assignment of classes to frequency ranks changes. Parentheses report RAP minus the corresponding baseline, calculated from the displayed values.

MethodAssignmentImageNet BaseNewHFood101 BaseNewH
CoOpRandom69.0761.2364.9184.4787.8386.11
+ RAPRandom74.57 (+5.50)66.43 (+5.20)70.26 (+5.35)88.50 (+4.03)89.23 (+1.40)88.86 (+2.75)
CoOpDifficulty72.6665.2568.7681.3581.7981.57
+ RAPDifficulty74.56 (+1.90)67.41 (+2.16)70.80 (+2.04)86.10 (+4.75)84.60 (+2.81)85.34 (+3.77)
CoOpDiversity73.7364.3668.7283.3582.3582.85
+ RAPDiversity74.76 (+1.03)64.90 (+0.54)69.48 (+0.76)87.20 (+3.85)86.19 (+3.84)86.69 (+3.84)
CoCoOpRandom72.3066.7069.3987.5390.3088.89
+ RAPRandom75.40 (+3.10)69.70 (+3.00)72.43 (+3.04)90.57 (+3.04)91.40 (+1.10)90.98 (+2.09)
CoCoOpDifficulty74.5269.7872.0788.0089.7888.88
+ RAPDifficulty75.18 (+0.66)70.43 (+0.65)72.73 (+0.66)90.37 (+2.37)90.62 (+0.84)90.49 (+1.61)
CoCoOpDiversity74.1468.0270.9588.7190.4289.56
+ RAPDiversity75.23 (+1.09)68.51 (+0.49)71.72 (+0.77)90.19 (+1.48)91.43 (+1.01)90.80 (+1.24)

RAP improves base and new accuracy in all eight additional comparisons. The H gains are 0.66-3.77 points for difficulty-based assignment and 0.76-3.84 for diversity-based assignment. We have added these frequency-assignment experiments to Section 5 and described the two controls in Section 3.

W3. Generality across architectures and prompt learners

A3. The additional experiments cover ResNet-50 and ViT-B/32 with both CoOp and CoCoOp. RAP improves base accuracy, new accuracy, and H in all four comparisons:

MethodBackboneBaseNewH
CoOpRN-5069.5954.0059.49
+ RAPRN-5073.00 (+3.41)55.97 (+1.97)62.20 (+2.71)
CoOpViT-B/3271.8460.4164.95
+ RAPViT-B/3274.43 (+2.59)64.65 (+4.24)68.83 (+3.88)
CoCoOpRN-5067.9661.2063.79
+ RAPRN-5071.10 (+3.14)61.80 (+0.60)65.33 (+1.54)
CoCoOpViT-B/3269.5065.7867.21
+ RAPViT-B/3274.75 (+5.25)70.64 (+4.86)72.47 (+5.26)

We also evaluated KgCoOp (Yao et al., CVPR 2023) with ViT-B/16 at ρ=64\rho=64, averaged over the six datasets:

MethodBaseNewHHeadMidTailGap (pp) ↓
KgCoOp71.2173.5872.3785.0976.5362.3622.73
+ RAP75.59 (+4.38)76.30 (+2.72)75.94 (+3.57)77.52 (-7.57)77.43 (+0.90)69.25 (+6.89)8.26 (-14.47)

KgCoOp gains 3.57 points in H and 6.89 in tail accuracy, with a 7.57-point decrease in head accuracy. We have added the backbone and KgCoOp results to Section 5.

AuthorsOpenReview ↗

Response to Reviewer EcQ9 (Part 2/2)

W4. Motivation for removing the rarity-dependent residual

A4. Rarity changes how the retained prompt parameters are learned. We optimize θ\theta and ΔV\Delta V jointly, and the gradient for θ\theta is evaluated at Vθ(x)+rcΔVV_\theta(x)+r_c\Delta V (Eq. 5). Removing the residual leaves the learned θ\theta unchanged, including CoCoOp's image-conditioning network. Eq. 5 explains this training effect; the inference ablation below tests whether removing the residual helps accuracy. We keep the trained parameters fixed and vary only α\alpha in Uctest(x;α)=Vθ⋆(x)+αrcΔV⋆U_c^{\mathrm{test}}(x;\alpha)=V_{\theta^\star}(x)+\alpha r_c\Delta V^\star. The test uses base classes, whose training frequencies are known, and averages the six datasets at ρ=64\rho=64. Here α=0\alpha=0 removes the residual and α=1\alpha=1 retains it fully.

Methodα\alphaBaseHeadMidTailGap (pp) ↓
CoOp0.0079.5185.7480.9668.9716.77
0.2579.3287.4281.1269.2018.23
0.5079.0588.8981.4965.4523.45
0.7578.4990.1080.7561.2128.89
1.0076.3690.5978.9255.7934.80
CoCoOp0.0079.0485.9379.5967.5218.41
0.2578.6187.4878.9165.4422.04
0.5077.7488.5877.8162.9225.66
0.7576.3589.3176.2059.7129.60
1.0074.4589.7273.8955.9733.75

Complete removal gives the highest base accuracy among the tested strengths: 79.51 versus 76.36 with full retention for CoOp, and 79.04 versus 74.45 for CoCoOp. Tail accuracy rises by 13.18 and 11.55 points, respectively, while head accuracy decreases. We have added the retention ablation in Section 5 and explained which parameters are retained in Section 4.

I would like to thank the authors for their detailed rebuttal. The responses have addressed most of my concerns, and I decide to maintain my positive score.

Comments on the paper

1 comment

Dear Reviewers,

The authors have submitted their responses to your comments. Please review their rebuttals and engage in further discussion, particularly regarding any remaining concerns, unresolved issues, or points requiring additional clarification.

Thank you for your time and continued contributions to the review process.

Regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.