- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Gradual Withdrawal, a training strategy for visual prompt tuning (VPT) that leverages task-specific knowledge from full fine-tuning (FT) without increasing the deployed model size. The method first fully fine-tunes a pretrained backbone on a downstream task and then trains visual prompts while gradually interpolating the backbone weights back to their original pretrained values. The final model retains only the original frozen backbone, learned prompts, and a classification head. The authors provide a theoretical analysis based on solution-path tracking to explain the benefit of gradual withdrawal. Experiments across diverse vision transformer backbones and downstream classification benchmarks demonstrate consistent improvements over conventional VPT and competitive prompt tuning methods. Ablation studies highlight the importance of the withdrawal schedule and task-specific FT updates, while additional analyses investigate the optimization landscape and attention patterns of the learned prompts.
Critical strengths
-
Novel and interesting approach. The paper introduces a simple yet interesting training strategy that leverages FT-derived adaptation to improve VPT performance while preserving the original deployment architecture.
-
Strong empirical results. The proposed method consistently outperforms direct VPT across diverse vision transformer backbones and downstream classification tasks, demonstrating its effectiveness across different pretraining paradigms and model scales.
-
Comprehensive ablation studies. The paper provides extensive comparisons with alternative training paths, including knowledge distillation, abrupt return, and shuffled withdrawal, effectively demonstrating the importance of the gradual withdrawal strategy.
Critical weaknesses
-
Insufficient motivation. The motivation for the proposed approach is not fully convincing. The authors argue that existing VPT methods cannot leverage the task-specific adaptation provided by FT, but it is unclear why this should be considered a fundamental limitation of VPT, given that freezing the backbone is an intentional design choice. Moreover, although the authors distinguish between prompt expressiveness and optimization, it is unclear whether optimization is a major bottleneck in conventional VPT or why FT-derived adaptation is a suitable way to address it. A clearer justification of these connections would strengthen the motivation.
-
Limited theoretical justification. The theoretical analysis in Section 4.2 provides a bound on the error when tracking a sequence of nearby solutions. However, this bound only shows that gradual withdrawal can track a solution path, without explaining why the final solution should outperform direct VPT. While the analysis provides a theoretical justification for gradual rather than abrupt withdrawal, it does not establish why following an FT-guided path leads to a better final solution.
-
Unclear rationale for FT guidance. The rationale for using FT as an optimization guide remains unclear. Table 2 shows that FT performs worse than direct VPT, yet its weight updates lead to substantial improvements in prompt performance. While this is an interesting empirical finding, the paper provides limited insight into why a weaker FT model offers useful guidance for learning a stronger prompt. Further analysis of what makes FT-derived updates beneficial would strengthen the proposed method.
-
Increased training cost. The proposed method requires FT for each downstream task before prompt optimization, substantially increasing the training cost compared to VPT. Although the method preserves deployment efficiency, the need for FT on every downstream task reduces one of the key practical advantages of PEFT, particularly when adapting to many tasks.
-
Lack of evaluation on VLMs. The experiments are limited to vision-only models, without evaluating VLMs such as CLIP. Given the widespread use of visual prompt in VLMs, this limits the demonstrated applicability of the proposed method.
Questions for authors
-
Why does a weaker FT provide an effective optimization path that leads to better performance than direct VPT? Given that direct VPT outperforms FT in Table 2, could the authors further explain what makes FT-derived guidance particularly effective? Additionally, VPT-to-VPT distillation baseline using the stronger direct VPT as the teacher would help assess whether similar improvements can be achieved through knowledge transfer from a better-performing teacher.
-
Figure 5 shows that gradual withdrawal produces different attention patterns compared to direct VPT. However, the analysis does not include the attention patterns of the FT. Could the authors provide a comparison with the FT to examine whether the observed changes reflect attention patterns learned during FT?
-
Generalization is important considerations in VLM-based prompt tuning, as highlighted by prior studies [1-3]. Since the proposed method relies on task-specific FT weights, it remains unclear whether its benefits extend to VLMs while preserving their generalizability.
[1] Read-only Prompt Optimization for Vision-Language Few-shot Learning, ICCV 2023
[2] Self-regulating Promptsː Foundational Model Adaptation without Forgetting, ICCV 2023
[3] Consistency-Guided Prompt Learning for Vision-Language Models, ICLR 2024