acceptodds
Under review as a conference paper at ICLR 2027

You Only Reprogram Once: Rethinking Prolonged Training for Visual Reprogramming

Abstract

Visual reprogramming is a parameter-efficient method for adapting pretrained models, yet its training can remain computationally expensive: even with a frozen backbone, visual prompts are often optimized through the full model for hundreds of epochs. Before changing what the pretrained model sees, we ask whether we are fully using what it already tells us. We find that modeling the full source response can already yield strong downstream predictions without prompt optimization. Motivated by this observation, we introduce You Only Reprogram Once (YORO), which constructs a downstream predictor from the frozen response space in a single forward-only traversal. Its Bayesian Discriminant Mapping (BDM) derives a covariance-aware affine mapping from streaming class statistics, requiring no backpropagation, optimizer updates, or repeated visits to the training set. When further input adaptation helps, YORO-FP optionally refines the visual prompt for 20 epochs. BDM also extends naturally to CLIP by treating attribute-prompt similarities as source responses. Across three full-data settings, YORO improves average accuracy over the strongest prior gradient-free mapping by 18.4–24.4%. On 16-shot CLIP, it raises the four-backbone average from 71.4% to 77.2%. YORO-FP provides further gains on selected tasks, while validation often retains the one-pass predictor. These results suggest a different default for visual reprogramming: read out the frozen response first, and optimize the input only when needed.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Model Reprogramming, Image Classification

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_g3LcOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the optimization problem in visual reprogramming, which involves adapting frozen source-pretrained models to a target domain using a small learnable pixel mask per sample. The method investigates whether prolonged visual-prompt optimization is necessary in this scenario. To this end, the authors propose YORO, which constructs a downstream classifier from the complete pretrained response vector using a single forward-only traversal of labeled training data. The method introduces a novel component, namely Bayesian Discriminant Mapping, which applies regularized linear discriminant analysis to class means and a shared covariance estimated through streaming statistics.

Regarding the experimental results, YORO covers both full-data visual reprogramming and 16-shot CLIP adaptation and reports substantial improvements over existing visual-reprogramming mappings, including an increase from 71.4% to 77.2% average accuracy over four CLIP backbones. The authors also report that timing experiments demonstrate a large reduction in adaptation cost compared with methods that optimize visual prompts for many epochs.

Critical strengths

The strengths of the paper can be summarized as follows:

  1. The paper addresses a sound and practically relevant problem in visual reprogramming. Although adapting to the target domain requires only a subset of parameters, training efficiency remains unoptimized since it still requires expensive backpropagation through a large frozen backbone. Thus, examining how much downstream information can already be recovered from its outputs is well motivated and brings a practical use case to the literature.
  2. The core methodology is coherent and clearly specified. The affine discriminant rule follows from the shared-covariance Gaussian model, and the streaming statistics recover the corresponding batch estimate without storing individual training responses.
  3. The evaluation and ablations cover diverse benchmark datasets, choice of backbones, input configurations, and supervision regimes. Some ablations are worth noting, such as the covariance ablations, which provide evidence for the effectiveness of regularization, and the mapping-update ablations, which examine different strategies for updating the classifier during prompt refinement. These analyses help readers understand the contribution of each design choice and the conditions under which additional prompt training would be beneficial.
Critical weaknesses

In my perspective, the paper contains no critical weaknesses apart from the fact that the efficiency experiments do not fully establish the computational advantage across the evaluated settings. As shown in Table 5, the training time is demonstrated to be substantially reduced in comparison to prolonged prompt optimization; however, the reported timing excludes the validation phase, although validation is required to select the covariance-shrinkage coefficient. I suggest that the paper can include more indicator metrics for efficiency, such as wall-clock adaptation time and training GPU usage. The efficient CLIP implementation also uses visual features to avoid constructing the full response covariance, leaving the computational cost under an output-only setting less clear. Reporting the complete efficiency metrics would provide stronger evidence for the efficiency claim.

Questions for authors

N/A

Reviewer_NH4COpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether visual reprogramming requires prolonged optimization of the input prompt. It proposes Bayesian Discriminant Mapping (BDM), which uses the full response vector of a frozen model to fit a regularized, LDA-style affine output mapping from streaming statistics in one pass. It also introduces YORO-FP, an optional prompt-refinement procedure that updates the prompt and tracks changes in the mapping. The paper evaluates YORO on full-data classification with frozen ResNet-18 and ResNeXt backbones, using padding and watermarking prompts, and on 16-shot CLIP classification across multiple datasets and backbones. The results suggest that YORO can provide competitive accuracy at lower adaptation cost. However, the strength of the comparative claims depends on whether prior-method results were obtained under the same input protocol, and the paper’s discussion of recent CLIP visual-reprogramming work could be more complete.

Critical strengths
  1. BDM uses the complete frozen-model response rather than only top-1 or selected top-K predictions, and fits its mapping analytically from a single pass. This directly addresses the paper’s question about reducing repeated prompt optimization.

  2. The paper evaluates both standard image classifiers and CLIP, and includes an optional prompt-refinement variant. This helps show where one-pass mapping may be useful and where additional prompt training can still matter.

Critical weaknesses
  1. The main comparisons are not sufficiently protocol-matched. Table 1 states that its baseline results are taken from Cai et al. (2024a), while YORO uses resized 192×192 images within a 224×224 canvas for padding. The paper does not establish that the copied baseline numbers were obtained under this resized-input protocol. Table 9 is more closely aligned with BLM’s legacy direct-padding setting, but its baseline results are also imported rather than rerun, and YORO does not surpass Deep in that comparison. Consequently, the broad superiority claims—especially over Deep—are not established by a controlled comparison.

  2. The related-work positioning for CLIP visual reprogramming is incomplete. The paper compares with DVP and AttrVR, but should discuss the more recent "Boosting Visual Reprogramming for CLIP with Dual Granularity Alignment, CVPR 2026". DGA uses a different mechanism, so it is not automatically a like-for-like baseline for BDM. Nonetheless, it is relevant to positioning YORO among recent methods for CLIP visual reprogramming, particularly for accuracy–compute tradeoffs.

Questions for authors
  1. For Table 1, were BLM, BLM+, Deep, and the other prior methods rerun with the stated 192×192 resize and 16-pixel border? If not, please clarify the exact input preprocessing and evaluation protocol used to obtain the copied results, and narrow the claims accordingly.

  2. For Table 9, can the author confirm that the baseline and YORO results use the same direct-padding geometry, data splits, pretrained checkpoints, training augmentations, and validation/model-selection protocol? Which of these settings are matched, and which differ?

  3. Could the author provide matched reruns of the key baselines—at minimum BLM, BLM+, and Deep—under the legacy and resized-padding protocols, using common splits and seeds? Per-seed results would help determine whether the reported differences are robust.

  4. How does YORO relate to DGA (CVPR 2026), which studies CLIP visual reprogramming using a different prompt-learning mechanism? A focused comparison or discussion of the differences in supervision, protocol, accuracy, and adaptation cost would clarify YORO’s position relative to recent work.

3 comments
AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer NH4C,

We wanted to briefly follow up on our response of November 12. Revised Table 1 now includes three-seed reruns of BLM, BLM+, and DeepLM using the same resized-padding geometry as YORO. Under this matched preprocessing, YORO exceeds the strongest baseline (DeepLM) by 6.2 and 3.4 percentage points on the two backbones. The original submission also included comparisons using matched input settings in Tables 2 and 9. We have also expanded the discussion of DGA.

We hope these additions address your concerns. We would be grateful if you could consider raising your score in light of the new evidence. We are also happy to provide any further clarification if needed.

Thank you again for your time and consideration.

Thank the authors for the response and the additional experiments. The extra experiments substantially address my main concern about the fairness of Table 1 comparison. Although the advantage over DeepLM is smaller than initially reported, YORO remains better on average under the revised setting. The clarification of Table 9 and the discussion of DGA also address my concerns.

I encourage the authors to clearly distinguish results under different preprocessing settings, document the implementation details, and clarify the timing boundaries in the comparison with DGA. Considering the new evidence and the practical value of the proposed single-pass approach, I raise my recommendation from weak reject to weak accept.

We thank the reviewer for the constructive feedback and for recognizing the practical value of our single-pass approach. In the revised manuscript, we will clearly distinguish results under different preprocessing settings, document the implementation details, and clarify the timing boundaries in the comparison with DGA. Thank you again for your time and careful assessment.

Reviewer_LdEJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether prolonged prompt optimization is necessary for visual reprogramming. The authors argue that existing methods do not fully utilize the class information available in the outputs of pretrained models. To address this issue, they propose YORO, a framework that constructs downstream predictors using Bayesian Discriminant Mapping (BDM). Specifically, BDM models the complete output responses of a frozen model and uses class means and covariance statistics to construct an affine discriminant mapping. This process requires only a single forward pass through the training dataset, without backpropagation or iterative optimization. The authors also introduce YORO-FP, which performs limited prompt refinement when further input adaptation is beneficial. Experiments on image classification and few-shot CLIP adaptation demonstrate improvements in classification accuracy and training efficiency.

Critical strengths

This paper is addresses a practical problem in visual reprogramming. Although existing approaches optimize only a small number of parameters, they often require many training epochs, resulting in considerable computational costs. Instead of focusing on accelerating prompt optimization, the authors investigate whether useful downstream information can already be extracted from the pretrained model's outputs. This is a simple but interesting perspective. By constructing the output mapping through a single forward-only traversal, YORO avoids repeated backpropagation and can be particularly useful when working with frozen backbones or models that only provide access to their outputs.

Another strength is that the proposed method is relatively simple and has a clear mathematical foundation. BDM is based on regularized linear discriminant analysis and constructs the output mapping using class means and a shared covariance matrix. Covariance shrinkage is further applied to improve numerical stability. Compared with previous mapping methods that rely mainly on Top-1 or Top-K class information, BDM can utilize the complete output response of the pretrained model. The authors also show that the required statistics can be obtained through streaming computation without storing all training responses. These properties make the approach easy to understand and potentially applicable to different frozen models.

The experimental evaluation is also relatively broad. The authors test YORO on multiple image classification datasets, different pretrained backbones, and few-shot CLIP adaptation tasks. The proposed method generally outperforms the main conventional mapping baselines. For example, in the 16-shot CLIP experiments, the average accuracy across four backbones increases from 71.4% for DVP to 77.2% for YORO. The paper also compares YORO with YORO-FP and shows that additional prompt optimization is not always necessary. Therefore, the experiments provide evidence not only for improved classification performance but also for the potential computational benefits of the proposed framework.

Critical weaknesses
  1. The methodological novelty appears somewhat limited. As the authors explicitly state in Section 3.2, BDM essentially applies classical regularized linear discriminant analysis to the output space of a pretrained model. The use of class means, shared covariance, and covariance shrinkage is already well established, and computing these statistics in a streaming manner is not a new mathematical result. Therefore, the main contribution seems to be the application of existing statistical techniques to visual reprogramming rather than the development of a fundamentally new learning mechanism. Although this application is useful, the paper should provide stronger evidence that BDM is preferable to other simple supervised mappings, such as ridge regression or linear classifiers, under matched input representations and training conditions.

  2. some experimental comparisons may not be fully controlled. Several baseline results in the main experiments are directly taken from previous publications instead of being reproduced under exactly the same conditions. In particular, the image resizing and input construction used in the padding setting are not fully consistent with those of some original baselines. Although Appendix Table 9 provides additional results under the legacy padding setting and still demonstrates improvements, the magnitude of the gains differs from the main results. Therefore, I suggest comparing the main methods using consistent preprocessing, input transformations, and computational budgets. This would help clarify whether the improvements are mainly due to the proposed output mapping or are partially influenced by differences in experimental configurations.

  3. YORO depends heavily on whether the frozen model's outputs already contain sufficient information for the downstream task. This assumption may not hold when the source and target domains are substantially different. For example, on SVHN, the one-pass YORO method performs considerably worse than prompt-optimized approaches in some settings, suggesting that output mapping alone may be insufficient for certain tasks. Although YORO-FP can address this issue to some extent, it also introduces additional training costs. The paper mainly reports the performance differences across tasks but provides limited analysis of how to determine in advance when prompt optimization is necessary. This limitation affects the practical usefulness of YORO as a general-purpose efficient adaptation framework.

Questions for authors

Since BDM is essentially based on regularized discriminant analysis, could the authors provide direct comparisons with ridge regression and linear classifiers using the same full output vectors and input settings? This would help clarify the specific advantages of BDM.

For datasets such as SVHN, where the target task differs substantially from the source task, is it possible to predict whether YORO-FP is necessary based on the initial output separability or validation performance? Could such a decision be made without significantly increasing computational costs?

Comments on the paper

1 comment

Dear Reviewers,

As the discussion deadline (November 18, AoE) approaches, please review the authors’ rebuttal if you have not already done so and post a follow-up confirming that you have read it and indicating whether your main concerns have been addressed or whether any issues remain.

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.