- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses checkpoint and layer selection for few-shot recognition when task-specific validation data are unavailable. It proposes gF1, the harmonic mean of training accuracy gains and retrieved-data accuracy preservation, for checkpoint and layer selection. The authors incorporate this criterion into a retrieval-augmented, stage-wise partial finetuning pipeline called VEST and evaluate its in-distribution and out-of-distribution performance across five benchmarks.
Critical strengths
-
The paper addresses a practical problem in few-shot recognition: checkpoint selection and hyperparameter tuning without task-specific validation data. The motivation is clear and relevant to realistic model adaptation scenarios.
-
The proposed gF1 criterion is simple and applicable to both checkpoint and layer selection. Incorporating it into a stage-wise finetuning pipeline provides a practical approach to balancing task adaptation and generalization.
-
The experiments report improvements in in-distribution accuracy and average out-of-distribution performance under several few-shot settings. The ablation studies and additional backbone experiments further support the effectiveness of the proposed design.
Critical weaknesses
-
The two-step layer-selection procedure does not necessarily preserve Pareto optimality over all candidate checkpoints. Section 3.3 first retains one checkpoint per layer configuration using configuration-specific reference points, and then recomputes gF1 among the retained checkpoints. Since the reference point changes, a discarded checkpoint can dominate the final selection in both training and retrieved-data accuracy. A comparison with joint selection over all checkpoints using a common reference point would help clarify the effect of the two-step procedure.
-
Eq. (1) computes its reference point from the minimum accuracies observed across checkpoints. Extending training can therefore change the scores and relative ranking of previously obtained checkpoints, even when the newly added checkpoints are not selected. It would be helpful to examine whether checkpoint and layer selection remain stable across different maximum training durations.
-
The contribution of gF1 within the complete VEST pipeline is not clearly isolated. Table 3 uses gF1 in every configuration and mainly evaluates changes in the finetuning strategy. Comparing selection criteria under the same pipeline and candidate configurations would help clarify how much of the final improvement comes from the validation rule.
Questions for authors
Please see the questions raised in the weaknesses.
3 comments
Part 2 of our rebuttal
The reviewer asks whether gF1 remains stable for checkpoints across different maximum training durations.
In the updated manuscript, we have provided a systematic sensitivity analysis of gF1-based checkpoint selection to varying training durations (Appendix Table 11). For convenience in reviewing, we copy relevant results below, along with the result by validation-free strategy (defaulting to the last checkpoint). The results demonstrate the robustness of our gF1 to varying training durations and its consistent gains over the validation-free alternative.
| epoch=20 | epoch=50 | epoch=100 | |||||||
|---|---|---|---|---|---|---|---|---|---|
| method | ID test | OOD test | avg. | ID test | OOD test | avg. | ID test | OOD test | avg. |
| validation-free | 72.42 | 60.12 | 66.27 | 72.32 | 59.33 | 65.83 | 72.58 | 58.61 | 65.60 |
| gF1 score | 72.83 | 61.03 | 66.93$^{\textcolor{Green}{+0.66}}$ | 72.58 | 60.94 | 66.76$^{\textcolor{Green}{+0.93}}$ | 72.80 | 60.86 | 66.83$^{\textcolor{Green}{+1.23}}$ |
The reviewer asks whether gF1 remains stable for layer selection across different reference points (e.g., perturbing the minimum accuracies) for gF1 calculation.
Changing the minimum reference values shifts the gF1 level sets and can alter which checkpoint is selected; yet, the underlying Pareto front of candidate checkpoints remains unchanged. We have conducted a sensitivity analysis w.r.t the ID/OOD performance of gF1-selected checkpoints with varying reference values. Specifically, using joint checkpoint and layer selection, we perturb the minimum reference values within 5 accuracy points, and re-run gF1-based selection. The results demonstrate that gF1 is highly robust to these perturbations and achieves remarkable performance gains compared with the previous strong method of full-finetuning (Liu, et al., 2025). We have added this analysis as Figure 6 in the updated manuscript, with complete numerical results in Appendix Table 12. We invite the reviewer to refer to them for visualization and in-depth analysis.
The reviewer requests a study of using different validation methods in Table 3 to validate the final improvement of applying gF1 in the VEST pipeline.
In the revised manuscript, we augmented Table 3 by adding Table 4, which compares gF1 against the validation-free strategy that defaults to selecting the last checkpoint. As the validation-free strategy lacks an inherent mechanism to determine the layer depth for finetuning, we adopt full finetuning in this comparison. For the reviewer's convenience, we reproduce Table 4 below. The results clearly isolate the contribution of gF1 and demonstrate the consistent gains of using gF1 across both training stages.
| Stage-1 | Stage-2 | accuracy | ||
|---|---|---|---|---|
| tuning | RA | tuning | ID test | OOD test |
| F | × | × | 72.32 | 59.33 |
| F | × | × | 72.58$^{\textcolor{Green}{+0.26}}$ | 60.94$^{\textcolor{Green}{+1.61}}$ |
| F | √ | × | 71.43 | 59.51 |
| F | √ | × | 72.89$^{\textcolor{Green}{+1.46}}$ | 63.28$^{\textcolor{Green}{+3.77}}$ |
| F | √ | F | 73.52 | 59.92 |
| F | √ | F | 74.49$^{\textcolor{Green}{+0.97}}$ | 63.58$^{\textcolor{Green}{+3.66}}$ |
Thank you for the detailed responses. The authors have addressed all my concerns regarding the Pareto optimality of the two-step layer-selection procedure, the stability of checkpoint selection across different training durations, and the contribution of gF1 within the VEST pipeline. I appreciate the authors' efforts in addressing these points and maintain my positive evaluation of the paper.
Part 1 of our rebuttal
We appreciate the reviewer’s constructive feedback, which helped us significantly improve the quality of our manuscript; we have uploaded a revised version with all changes highlighted in blue. The reviewer recognized the strengths of our work, including our clear motivation to address a practical problem in realistic model adaptation scenarios, our simple and applicable validation method, and our comprehensive experiments and ablation studies. Below, we address each of the reviewer’s concerns and questions. Should any further clarification be needed, we would be happy to provide additional information.
Thank you for this insightful question. We agree that verifying whether the two-step layer-selection procedure preserves Pareto optimality is critical. In the revised manuscript, we address this theoretically and empirically:
Theoretical guarantees. We rigorously demonstrate that the two-step procedure preserves and selects globally Pareto-optimal points
Theorem 1 (page 4): maximizing the harmonic mean (i.e., the F1 score) identifies a Pareto-optimal point.
[R1, R2]: Shifting reference baselines (e.g., minimum accuracy subtraction used in gF1) does not alter Pareto front. Consequently, our gF1 selects Pareto-optimal points.
Theorem 2 & Corollary 1 (page 5): The checkpoints selected by gF1, either through the two-step procedure or the joint selection paradigm, are Pareto-optimal.
Empirical comparison. In the revised manuscript, we have updated Figure 5 to compare the results of the two-step procedure and the joint selection paradigm. With both paradigms, our gF1 identifies top-4 (i.e., k=4) layers to finetune, although selecting different checkpoints due to different reference baselines in calculating gF1 scores. Below is a synthesis of their comparison based on Figure 5 and Appendix Table 12:
Practical Trade-offs. We have added a discussion paragraph at Line298 analyzing the trade-offs between the two paradigms. In brief, both yield Pareto-optimal points with our gF1, although their selected layers and checkpoints can differ due to variations in the reference values used for gF1 calculation. Yet, joint selection requires retaining all checkpoints while the two-step method only needs to save one checkpoint for each k. Therefore, the two-step paradigm offers superior storage efficiency in practice.
reference:
In the updated manuscript, we have provided a systematic sensitivity analysis of gF1-based checkpoint selection to varying training durations (Appendix Table 11). For convenience in reviewing, we copy relevant results below, along with the result by validation-free strategy (defaulting to the last checkpoint). The results demonstrate the robustness of our gF1 to varying training durations and its consistent gains over the validation-free alternative.