- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Reparameterized Inter-Class Visual Reprogramming (RVP) for few-shot adaptation of frozen CLIP models. RVP jointly learns a visual prompt, within-class text-description aggregation weights, and a residual linear transformation across class logits. Aggregating descriptions within classes before combining evidence across classes yields a structured classifier parameterization that can be folded into a single linear classifier at inference. Experiments cover 11 datasets and four CLIP backbones, primarily under 16-shot supervision, with substantial gains on fine-grained recognition tasks.
Critical strengths
-Although linear at inference, the training-time classifier parameterization constrains how descriptions contribute to target classes, providing a reasonable few-shot inductive bias. In Table 4, removing the inter-class matrix reduces average accuracy from 82.7% to 77.9%, while uniform intra-class aggregation retains 82.6%. These results support inter-class adaptation and suggest that learned description weights contribute little to the reported average accuracy.
-RVP improves average accuracy over DVP from 79.7% to 82.7% with ViT-B/16 and from 66.0% to 72.3% with RN50, with substantial gains on Aircraft and Cars. Table 12 controls text descriptions and restricts DVP to one visual prompt; Table 8 includes broader CLIP adaptation baselines. RVP requires one image-encoder forward pass, and its output mappings admit exact folding, offering a useful accuracy–efficiency trade-off.
Critical weaknesses
-Figure 1 shows overlapping descriptions and concentrated feature energy, but does not establish that dominant directions encode harmful shared semantics or that low-energy directions contain discriminative information. The appendix’s correlations and matrix analyses provide support, but do not directly demonstrate that the learned improves classification by suppressing shared semantics.
-Linear Probe and Attribute & LP are jointly trained with the visual prompt, but it is unclear whether they use matched text-based initialization, comparable regularization, and matched tuning budgets. These differences could confound attribution of the gains to the structured parameterization.
Questions for authors
-How are Linear Probe and Attribute & LP in Table 4 initialized? Do they produce the same initial logits as RVP?
-Building on the uniform-aggregation ablation, let be the fixed classifier formed from average text prototypes. Is there a comparison of with , initializing both and to zero and using identical visual-prompt initialization? If not, could the authors provide one to isolate structural benefits beyond text-based initialization?
-Beyond Appendix B.4’s correlations and matrix analyses and Appendix C.1’s conditional theoretical results, is there direct evidence that the transformation suppresses harmful shared components while preserving discriminative information? Directional projection experiments or before/after discriminability analyses could help. Otherwise, could shared-semantic suppression be framed more explicitly as a plausible explanation rather than an established mechanism?