- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses the optimization problem in visual reprogramming, which involves adapting frozen source-pretrained models to a target domain using a small learnable pixel mask per sample. The method investigates whether prolonged visual-prompt optimization is necessary in this scenario. To this end, the authors propose YORO, which constructs a downstream classifier from the complete pretrained response vector using a single forward-only traversal of labeled training data. The method introduces a novel component, namely Bayesian Discriminant Mapping, which applies regularized linear discriminant analysis to class means and a shared covariance estimated through streaming statistics.
Regarding the experimental results, YORO covers both full-data visual reprogramming and 16-shot CLIP adaptation and reports substantial improvements over existing visual-reprogramming mappings, including an increase from 71.4% to 77.2% average accuracy over four CLIP backbones. The authors also report that timing experiments demonstrate a large reduction in adaptation cost compared with methods that optimize visual prompts for many epochs.
Critical strengths
The strengths of the paper can be summarized as follows:
- The paper addresses a sound and practically relevant problem in visual reprogramming. Although adapting to the target domain requires only a subset of parameters, training efficiency remains unoptimized since it still requires expensive backpropagation through a large frozen backbone. Thus, examining how much downstream information can already be recovered from its outputs is well motivated and brings a practical use case to the literature.
- The core methodology is coherent and clearly specified. The affine discriminant rule follows from the shared-covariance Gaussian model, and the streaming statistics recover the corresponding batch estimate without storing individual training responses.
- The evaluation and ablations cover diverse benchmark datasets, choice of backbones, input configurations, and supervision regimes. Some ablations are worth noting, such as the covariance ablations, which provide evidence for the effectiveness of regularization, and the mapping-update ablations, which examine different strategies for updating the classifier during prompt refinement. These analyses help readers understand the contribution of each design choice and the conditions under which additional prompt training would be beneficial.
Critical weaknesses
In my perspective, the paper contains no critical weaknesses apart from the fact that the efficiency experiments do not fully establish the computational advantage across the evaluated settings. As shown in Table 5, the training time is demonstrated to be substantially reduced in comparison to prolonged prompt optimization; however, the reported timing excludes the validation phase, although validation is required to select the covariance-shrinkage coefficient. I suggest that the paper can include more indicator metrics for efficiency, such as wall-clock adaptation time and training GPU usage. The efficient CLIP implementation also uses visual features to avoid constructing the full response covariance, leaving the computational cost under an output-only setting less clear. Reporting the complete efficiency metrics would provide stronger evidence for the efficiency claim.
Questions for authors
N/A
Follow-up on Our Rebuttal
Dear Reviewer NH4C,
We wanted to briefly follow up on our response of November 12. Revised Table 1 now includes three-seed reruns of BLM, BLM+, and DeepLM using the same resized-padding geometry as YORO. Under this matched preprocessing, YORO exceeds the strongest baseline (DeepLM) by 6.2 and 3.4 percentage points on the two backbones. The original submission also included comparisons using matched input settings in Tables 2 and 9. We have also expanded the discussion of DGA.
We hope these additions address your concerns. We would be grateful if you could consider raising your score in light of the new evidence. We are also happy to provide any further clarification if needed.
Thank you again for your time and consideration.