acceptodds
Under review as a conference paper at ICLR 2027

Reprogramming Vision-Language Models via Structured Prompt Reparameterization

Abstract

Visual reprogramming adapts pretrained models to downstream tasks by modifying their input and output interfaces while keeping the backbone fixed. In vision-language models, existing methods mainly rely on intra-class prompt aggregation and do not explicitly model relationships among classes. However, fine-grained categories often exhibit highly overlapping attribute descriptions and strong inter-class correlation in the text embedding space, where discriminative cues lie in subtle low-variance components. We propose Reparameterized Inter-Class Visual Reprogramming (RVP), a structured framework that aggregates multiple text prompts within each class and applies residual correction across classes. We also show that CLIP-based visual reprogramming with input-independent linear output aggregation can be expressed as a linear mapping from frozen image embeddings to downstream logits, and use this view to design a structured reparameterization that models shared semantic components and class-specific differences. RVP uses only a single visual prompt and can be reparameterized at inference into a frozen backbone followed by a linear classifier, incurring nearly zero computational overhead. Across 11 few-shot classification benchmarks and four CLIP backbones, RVP consistently improves over prior visual reprogramming methods with comparable or better inference efficiency.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision language models, model reprogramming

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qsUEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Reparameterized Inter-Class Visual Reprogramming (RVP) for few-shot adaptation of frozen CLIP models. RVP jointly learns a visual prompt, within-class text-description aggregation weights, and a residual linear transformation across class logits. Aggregating descriptions within classes before combining evidence across classes yields a structured classifier parameterization that can be folded into a single linear classifier at inference. Experiments cover 11 datasets and four CLIP backbones, primarily under 16-shot supervision, with substantial gains on fine-grained recognition tasks.

Critical strengths

-Although linear at inference, the training-time classifier parameterization constrains how descriptions contribute to target classes, providing a reasonable few-shot inductive bias. In Table 4, removing the inter-class matrix reduces average accuracy from 82.7% to 77.9%, while uniform intra-class aggregation retains 82.6%. These results support inter-class adaptation and suggest that learned description weights contribute little to the reported average accuracy.

-RVP improves average accuracy over DVP from 79.7% to 82.7% with ViT-B/16 and from 66.0% to 72.3% with RN50, with substantial gains on Aircraft and Cars. Table 12 controls text descriptions and restricts DVP to one visual prompt; Table 8 includes broader CLIP adaptation baselines. RVP requires one image-encoder forward pass, and its output mappings admit exact folding, offering a useful accuracy–efficiency trade-off.

Critical weaknesses

-Figure 1 shows overlapping descriptions and concentrated feature energy, but does not establish that dominant directions encode harmful shared semantics or that low-energy directions contain discriminative information. The appendix’s correlations and matrix analyses provide support, but do not directly demonstrate that the learned EE improves classification by suppressing shared semantics.

-Linear Probe and Attribute & LP are jointly trained with the visual prompt, but it is unclear whether they use matched text-based initialization, comparable regularization, and matched tuning budgets. These differences could confound attribution of the gains to the structured parameterization.

Questions for authors

-How are Linear Probe and Attribute & LP in Table 4 initialized? Do they produce the same initial logits as RVP?

-Building on the uniform-aggregation ablation, let W0W_0 be the fixed classifier formed from average text prototypes. Is there a comparison of W=W0(I+E)W=W_0(I+E) with W=W0+ΔWW=W_0+\Delta W, initializing both EE and ΔW\Delta W to zero and using identical visual-prompt initialization? If not, could the authors provide one to isolate structural benefits beyond text-based initialization?

-Beyond Appendix B.4’s correlations and matrix analyses and Appendix C.1’s conditional theoretical results, is there direct evidence that the transformation suppresses harmful shared components while preserving discriminative information? Directional projection experiments or before/after discriminability analyses could help. Otherwise, could shared-semantic suppression be framed more explicitly as a plausible explanation rather than an established mechanism?

Reviewer_9HHgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RVP for few-shot CLIP visual reprogramming. It learns within-class attribute aggregation and a residual inter-class logit transformation, both of which can be folded into a linear classifier at inference. The method improves over prior visual-reprogramming baselines on 11 datasets and four CLIP visual backbones.

Critical strengths

[+]: The problem formulation is meaningful. independent within-class prompt aggregation can leave cross-class ambiguity unresolved.

[+]: Results are extensive across datasets, backbones, and shot counts. The inter-class module ablation is substantial.

[+]: The inference folding is exact and provides a credible efficiency advantage over multi-forward-pass DVP variants.

[+]: This paper is well-organized and easy-to-follow.

Critical weaknesses

[-]: Novelty is really limited. The main novelty is a structured residual logit mapping, which is close to existing residual-classifier, label-mapping, and logit-correction ideas. Matched comparisons or discussions to these methods are missing.

[-]: Claims that E suppresses shared semantics are stronger than the evidence. The correlation analysis uses only 11 datasets, and the theory establishes existence under assumptions rather than properties of the learned E.

[-]: Baseline numbers are largely inherited from prior work and lack standard deviations; Also latency is only discussed at FGVC Aircraft.

Questions for authors

See weaknesses above.

Reviewer_x9RiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Authors propose visual reprogramming to improve fine-grained classification of visual tasks - they propose Reparameterized Inter-Class Visual Reprogramming (RVP), a method for few-shot adaptation of frozen CLIP models. Prior visual reprogramming methods aggregate multiple text attribute descriptions within each class independently but ignore relationships between classes a limitation in fine-grained settings where descriptions for different classes overlap heavily such as airplanes.

RVP addresses this with three contributions. First, it shows that CLIP-based visual reprogramming with a fixed prompt reduces to a linear mapping from image embeddings to class logits, unifying prior aggregation strategies. Second, it introduces a structured two-stage output head: a learnable per-class weighting over attribute descriptions, followed by a residual inter-class correction matrix initialized to zero. Third, these components can be folded exactly into a single linear classifier at inference, so the method runs at the same cost as a single-prompt baseline despite modeling class relationships during training.

Authors validate results over 11 benchmarks with thorough ablations (borrowing from other works in the domain) and showcase performance across various CLIP backbones at 16-shot classification.

Critical strengths

Authors RVP approach shows performance notable gains from prior SOTA work in the literature with notable improvements on the main 11 datasets, across various backbones and ablation studies. Compared to older SOTA, such as DVP where accuracy came at the cost of multiple image-encoder passes and roughly 2.4× more parameters. RVP is able to match the single-pass cost and parameter count of simpler baselines while reporting higher accuracy. 

Writing is elegant, and presented in a clear and well thought out fashion.

Critical weaknesses

Dataset coverage can be more thorough, current datasets are small/more niche and saturated, such as Caltech/Flowers/Pets (though admittedly the baseline in prior works). Would be a stronger paper including larger datasets such as ImageNet, or iNaturalist for additional fine-grained visual classification tasks.

Novelty and effect of the approach is limited to older families and classes of models. Though there are invaluable and interesting findings and theorems and analysis on the approach, the outcome feels more academic, then necessarily practical, as more modern stronger encoders appear to significantly diminish the impact and effect of RVP.

Questions for authors

1/ Would be great to address for authors less familiar w/ the domain - why or why not larger or more modern fine-grained visual classification dataset are not applicable (in the fashion of your appendix/discussion on foods/euro)

2/ Additional information on the practical use-case that this work may be able to address, would empower the paper from appearing more academic in nature (while very interesting). For example, how to address the diminishing improvements on larger encoders?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.