- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies prompt learning for vision-language models under class-imbalanced supervision. The authors introduce TailPromptBench, which evaluates long-tailed recognition while keeping the total number of training samples fixed. They also propose RAP, which adds a shared residual to the prompt during training, with its contribution determined by class frequency. The residual is removed during inference.
Critical strengths
-
I appreciate the fixed-budget evaluation protocol. Keeping the total number of training samples unchanged makes it easier to study the effect of class-imbalance without introducing differences in training-set size. The evaluation of both tail-class recognition and base-to-new generalization is also useful.
-
The proposed method is simple and introduces little additional training overhead. The results show consistent improvements over CoOp and CoCoOp under severe imbalance. The experiments with shuffled and constant rarity coefficients also provide some evidence that class-frequency information contributes to these improvements.
Critical weaknesses
-
My main concern is the lack of comparisons with existing methods specifically developed for long-tailed VL learning. The main experiments only compare RAP with CoOp and CoCoOp, neither of which was designed to handle class imbalance. The paper discusses more recent methods, including Candle and LAMP, but does not provide experimental comparisons with them. As a result, the current experiments show that rarity-aware prompting improves standard prompt learning, but they do not establish its advantage over existing imbalance-aware approaches.
-
The methodological novelty is somewhat limited. The main idea is to scale a shared residual prompt according to class frequency, which is a relatively straightforward modification. I am not convinced that this is substantially different from introducing frequency-dependent adjustments during optimization. Comparisons with standard class reweighting or logit adjustment would help clarify the contribution.
-
The motivation for removing the rarity-dependent residual at inference is understandable, since class frequencies are unavailable for unseen categories. However, the paper does not sufficiently investigate why discarding this residual leads to better generalization. I would also like to see a more detailed analysis of the trade-off between head- and tail-class accuracy, since Figure 5 shows that tail-class improvements often come with noticeable decreases in head-class performance.
Questions for authors
-
Can the authors compare RAP with recent LT VL adaptation methods, such as Candle and LAMP, under the same TailPromptBench protocol? This is particularly important for assessing the contribution of the proposed method.
-
How does RAP compare with simpler imbalance-handling techniques, such as class-balanced loss, class reweighting, or logit adjustment?
-
Have the authors evaluated alternatives to completely removing the residual at inference? For example, what happens if a fixed residual coefficient is used for all test classes?
2 comments
Comment
Some of my concerns have been addressed, so I will raise my score to 3.
Response to Reviewer bem3
We thank the reviewer for the actionable suggestions on long-tailed baselines and inference-time residuals.
W1 & Q1. Comparisons with long-tailed VLM methods
A1. We have added direct comparisons with Candle and LAMP on the same TailPromptBench splits and fixed training-image budget. The table reports six-dataset averages at ρ=64, with H averaged over dataset-level harmonic means. We also include class-balanced loss (CB, β=0.99) and logit adjustment (LA, τ=1.0), applied to both prompt learners and combined with RAP.
CoOp+RAP and CoCoOp+RAP achieve mean H of 75.18 and 74.71, respectively, compared with 73.67 for Candle and 73.81 for LAMP. We have added these comparisons and their experimental settings to Section 5.
W2 & Q2. Difference from class reweighting and logit adjustment
A2. Class-balanced loss changes the weight of a training example; logit adjustment changes the class scores. RAP changes the text representation used to compute those scores. Each candidate class receives the context Vθ(x)+rcΔV before the frozen text encoder. Rarity therefore affects the representations of both the positive class and the negative classes in the softmax. The original prompt and shared residual are trained jointly; inference retains the learned original prompt. The table above compares these choices directly: RAP has higher H than either CB or LA on both learners, and CB provides a further gain when combined with RAP. We have made the distinction from loss weighting and logit correction explicit in Section 4.
W3 & Q3. Alternatives to removing the residual
A3. We have not evaluated a non-zero coefficient shared by all test classes, Vθ⋆(x)+γΔV⋆. Our available experiment varies the retained training residual, Vθ⋆(x)+αrcΔV⋆, using the same trained checkpoints on base classes. Here α is shared but rc still varies by class. The table averages the six datasets at ρ=64:
Removal gives the highest base accuracy in this sweep. Full retention raises head accuracy but reduces base accuracy from 79.51 to 76.36 for CoOp and from 79.04 to 74.45 for CoCoOp; tail accuracy falls by 13.18 and 11.55 points. After removal, inference uses the jointly trained original prompt, including CoCoOp's conditioning network. We have added the residual-retention sweep and its precise inference rule to Section 5.
W3. Analysis of the trade-off between head- and tail-class accuracy
A3 (continued). For CoOp, the default is not the best head-tail choice in our sweep: λΔ=1 improves both head and tail accuracy over λΔ=0.1. The latter was selected for base-to-new H. For CoCoOp, the default λΔ=0.001 lies on the observed Pareto frontier; the alternatives that improve head accuracy lose tail accuracy. We have added the head-tail Pareto analysis in Appendix G and discussed it in Section 5.