acceptodds
Under review as a conference paper at ICLR 2027

Can Visual Prompts Inherit Full Fine-Tuning?

Abstract

Pretrained vision transformers have demonstrated strong transfer ability and are increasingly shared across many downstream tasks. This trend has driven growing interest in parameter-efficient adaptation, which aims to adapt one frozen backbone to each task with only a few task-specific parameters. Visual prompt tuning (VPT) is a representative approach that learns a small set of prompt tokens and a classifier for each task. However, existing VPT methods learn prompts only on the frozen pretrained backbone and therefore cannot use the task-specific adaptation that full fine-tuning (FT) provides. These limitations leave compact prompts well below the accuracy their architecture can reach. Therefore, this paper proposes gradual withdrawal. Specifically, gradual withdrawal first fine-tunes a copy of the backbone on the target task, and then trains visual prompts while smoothly returning the backbone to its original pretrained weights. The deployed model is exactly that of direct VPT: the original backbone plus a prompt and a classifier per task. Crucially, this ordered return lets the prompt track a sequence of nearby solutions, as our local tracking analysis explains. On supervised ViT-B/16, gradual withdrawal raises mean accuracy from 69.4% to 76.2% across 19 VTAB-1k tasks and from 89.1% to 92.3% across five fine-grained classification tasks, although the FT model itself scores only 65.6% and 88.5%. Extensive experiments show that gradual withdrawal outperforms matched distillation, abrupt-return, and shuffled-path controls, with gains that hold across eight backbones spanning 86M to 1.1B parameters. More broadly, these results establish the training path as a new axis of parameter-efficient adaptation: a compact model can learn from far more than it deploys, so better training alone can make it stronger.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Representation Learning, Vision Foundation Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qeQnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Gradual Withdrawal, a training strategy for visual prompt tuning (VPT) that leverages task-specific knowledge from full fine-tuning (FT) without increasing the deployed model size. The method first fully fine-tunes a pretrained backbone on a downstream task and then trains visual prompts while gradually interpolating the backbone weights back to their original pretrained values. The final model retains only the original frozen backbone, learned prompts, and a classification head. The authors provide a theoretical analysis based on solution-path tracking to explain the benefit of gradual withdrawal. Experiments across diverse vision transformer backbones and downstream classification benchmarks demonstrate consistent improvements over conventional VPT and competitive prompt tuning methods. Ablation studies highlight the importance of the withdrawal schedule and task-specific FT updates, while additional analyses investigate the optimization landscape and attention patterns of the learned prompts.

Critical strengths
  1. Novel and interesting approach. The paper introduces a simple yet interesting training strategy that leverages FT-derived adaptation to improve VPT performance while preserving the original deployment architecture.

  2. Strong empirical results. The proposed method consistently outperforms direct VPT across diverse vision transformer backbones and downstream classification tasks, demonstrating its effectiveness across different pretraining paradigms and model scales.

  3. Comprehensive ablation studies. The paper provides extensive comparisons with alternative training paths, including knowledge distillation, abrupt return, and shuffled withdrawal, effectively demonstrating the importance of the gradual withdrawal strategy.

Critical weaknesses
  1. Insufficient motivation. The motivation for the proposed approach is not fully convincing. The authors argue that existing VPT methods cannot leverage the task-specific adaptation provided by FT, but it is unclear why this should be considered a fundamental limitation of VPT, given that freezing the backbone is an intentional design choice. Moreover, although the authors distinguish between prompt expressiveness and optimization, it is unclear whether optimization is a major bottleneck in conventional VPT or why FT-derived adaptation is a suitable way to address it. A clearer justification of these connections would strengthen the motivation.

  2. Limited theoretical justification. The theoretical analysis in Section 4.2 provides a bound on the error when tracking a sequence of nearby solutions. However, this bound only shows that gradual withdrawal can track a solution path, without explaining why the final solution should outperform direct VPT. While the analysis provides a theoretical justification for gradual rather than abrupt withdrawal, it does not establish why following an FT-guided path leads to a better final solution.

  3. Unclear rationale for FT guidance. The rationale for using FT as an optimization guide remains unclear. Table 2 shows that FT performs worse than direct VPT, yet its weight updates lead to substantial improvements in prompt performance. While this is an interesting empirical finding, the paper provides limited insight into why a weaker FT model offers useful guidance for learning a stronger prompt. Further analysis of what makes FT-derived updates beneficial would strengthen the proposed method.

  4. Increased training cost. The proposed method requires FT for each downstream task before prompt optimization, substantially increasing the training cost compared to VPT. Although the method preserves deployment efficiency, the need for FT on every downstream task reduces one of the key practical advantages of PEFT, particularly when adapting to many tasks.

  5. Lack of evaluation on VLMs. The experiments are limited to vision-only models, without evaluating VLMs such as CLIP. Given the widespread use of visual prompt in VLMs, this limits the demonstrated applicability of the proposed method.

Questions for authors
  1. Why does a weaker FT provide an effective optimization path that leads to better performance than direct VPT? Given that direct VPT outperforms FT in Table 2, could the authors further explain what makes FT-derived guidance particularly effective? Additionally, VPT-to-VPT distillation baseline using the stronger direct VPT as the teacher would help assess whether similar improvements can be achieved through knowledge transfer from a better-performing teacher.

  2. Figure 5 shows that gradual withdrawal produces different attention patterns compared to direct VPT. However, the analysis does not include the attention patterns of the FT. Could the authors provide a comparison with the FT to examine whether the observed changes reflect attention patterns learned during FT?

  3. Generalization is important considerations in VLM-based prompt tuning, as highlighted by prior studies [1-3]. Since the proposed method relies on task-specific FT weights, it remains unclear whether its benefits extend to VLMs while preserving their generalizability.

    [1] Read-only Prompt Optimization for Vision-Language Few-shot Learning, ICCV 2023

    [2] Self-regulating Promptsː Foundational Model Adaptation without Forgetting, ICCV 2023

    [3] Consistency-Guided Prompt Learning for Vision-Language Models, ICLR 2024

Reviewer_3MkVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes gradual withdrawal, a new training procedure for visual prompt tuning (VPT). The method first prepares a pretrained model θ0\theta_0 and a model θFT\theta_{FT} fine-tuned on the task data. It then trains learnable prompt tokens and a classifier on an interpolated model θ(α)\theta(\alpha) constructed from these two models. The interpolation coefficient α\alpha changes dynamically over three training phases: warm-up, withdrawal, and final. At deployment, only θ0\theta_0, the prompts, and the classifier are used, so the deployed model is identical to that of standard VPT. Experiments on image classification tasks show that gradual withdrawal outperforms prompt-tuning baselines and, in some cases, even full fine-tuning.

Critical strengths
  • S1. The idea of exploiting the weights of a fine-tuned model, which existing VPT studies have not utilized, is novel, and gradual withdrawal shows better experimental results than the compared methods. In image classification experiments, gradual withdrawal achieves higher accuracy than six prompt-tuning methods. Moreover, gradual withdrawal surpasses the accuracy of fully fine-tuned models built on backbones pretrained with supervised ImageNet-21K or DINOv2. Its deployment cost is the same as that of standard VPT.

  • S2. The paper is well organized and easy to read. The method is explained visually with figures, which makes it easy to understand. In addition to the main experiments, the paper provides extensive ablations and analyses.

Critical weaknesses
  • W1. Training cost and choice of baselines. Because gradual withdrawal requires the weights of a fully fine-tuned model for every task, its training cost is higher than that of full fine-tuning alone and roughly twice or more than that of standard VPT. Although deployment remains parameter-efficient, the method gives up the training-time efficiency that is a key advantage of PEFT. Therefore, in addition to PEFT methods, comparisons with full-fine-tuning-based methods such as LP-FT [1] and WiSE-FT [2] under an equal training budget are necessary. WiSE-FT is particularly relevant, since it also uses models interpolated between the pretrained and fine-tuned weights. Nevertheless, the paper only cites WiSE-FT and does not compare against it.

  • W2. Evidence for the claim “A weaker predictor can be a useful guide.” The paper claims that “A weaker predictor can be a useful guide,” but this claim rests only on the accuracy of the endpoint θFT\theta_{FT}. The paper does not report the accuracy of the intermediate interpolated models θ(α)\theta(\alpha) on which the prompts are actually trained, nor that of the prompted θFT\theta_{FT} after warm-up. If these models are accurate, the performance gain may simply result from learning on strong interpolated models, and the interpretation of the claim could change.

[1] Kumar, Ananya, Raghunathan, Aditi, Jones, Robbie, Ma, Tengyu, and Liang, Percy. Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution. Retrieved from https://par.nsf.gov/biblio/10337813. International Conference on Learning Representations .

[2] Wortsman, Mitchell, et al. "Robust fine-tuning of zero-shot models." 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2022.

Questions for authors
  • Q1. Could you report the accuracy of the intermediate interpolated models θ(α)\theta(\alpha) and the prompted θFT\theta_{FT} after warm-up? If these exceed direct VPT, how should the claim “A weaker predictor can be a useful guide” be interpreted?

  • Q2. Could you provide comparisons with WiSE-FT?

Reviewer_FBRVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes gradual withdrawal, a training strategy for VPT that preserves the original pretrained backbone at deployment. It first fully fine-tunes a copy of the backbone on the target task. During subsequent prompt training, the backbone is progressively interpolated from the fine-tuned weights back to the original pretrained weights following a cosine schedule, while only the visual prompts and classifier are optimized. Finally, the prompts and classifier are further refined on the original frozen backbone. The deployed model therefore retains the same architecture and parameter count as standard VPT.

Critical strengths

The proposed method is well-motivated, conceptually simple, and technically sound. Its key novelty lies in decoupling the training trajectory from the deployed backbone, improving prompt learning without increasing deployment complexity.

The paper provides solid experimental validation through teacher-access controls, path-order comparisons, delta-source controls, training-phase ablations, and prompt-capacity studies, yielding several interesting insights into the effectiveness of the proposed approach.

The method achieves substantial improvements on established transfer learning benchmarks, with consistent gains across different backbone architectures, pretraining strategies, and model scales.

Critical weaknesses

Theory flaw: The mechanistic conclusions are too strong with less evidence supported: The main text uses the loss-landscape visualization to conclude that withdrawal reaches a different, flatter basin that direct optimization misses. But Appendix claimed that the loss-landscape and Hessian analyses are limited to two-dimensional slices, which cannot establish the geometry of the full parameter space.

The theoretical analysis assumes full-batch gradient descent and local strong convexity, while the experiments use adamw, learning-rate scheduling, dropout, and weight decay. It remains unclear whether the theoretical guarantees extend to these practical settings.

Questionable equal-compute comparison: Although Appendix F reports improvements under an equal 0.5 GPU-hour budget, the compute allocation and VPT restart configuration lack sufficient detail, and results at additional budgets are needed to establish a consistent accuracy–compute advantage.

Questions for authors

Where is the numerical results of label-scale evaluations on ImageNet-1K, Places365, and iNat2018, together with WILDS evaluations on iWildCam and FMoW listed in Table 8?

Which shuffled schedule produced Table 2? Does it permute the primary cosine states or the linear states from Table 4?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.