acceptodds
Under review as a conference paper at ICLR 2027

Enabling Validation for Robust Few-Shot Recognition

Abstract

A key challenge in Few-Shot Recognition (FSR) is the scarcity of labeled task-specific data for training. Contemporary methods mitigate this challenge by exploiting pretrained Vision-Language Models (VLMs) for their learned transferable features. Some recent ones augment the scarce data by retrieving relevant yet out-of-distribution (OOD) examples from external data sources. However, they largely overlook an equally fundamental challenge: the absence of validation data for model selection and hyperparameter tuning. As a result, the adapted VLMs overfit the few-shot training data and generalize poorly to OOD test data. We show that retrieved data provide a natural source of validation, but their direct use creates a fundamental paradox. Since the retrieved samples are inherently OOD w.r.t the few-shot ID training data, adapting a VLM to the ID data constantly degrades its performance on the retrieved data. Consequently, conventional validation relying on the accuracy of retrieved data favors no adaptation and prevents improved generalization. To resolve this dilemma, we formulate validation as a multi-objective optimization (MOO) problem between performance gains on the few-shot ID data and the performance preservation on the retrieved data. We introduce a simple yet effective validation strategy by choosing an operating point on the Pareto front of the MOO problem. This strategy enables hyperparameter tuning, which had been unrealistically done by using test data, thereby mitigating overfitting and improving generalization. We integrate it into a stage-wise finetuning pipeline, termed *Validation-Enabled Stage-wise Tuning (VEST)*. Extensive experiments on five established benchmarks show that VEST consistently outperforms existing approaches on both ID and OOD test data.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
few-shot recognition, validation, vision-language model, out-of-distribution test data

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_G8foOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses checkpoint and layer selection for few-shot recognition when task-specific validation data are unavailable. It proposes gF1, the harmonic mean of training accuracy gains and retrieved-data accuracy preservation, for checkpoint and layer selection. The authors incorporate this criterion into a retrieval-augmented, stage-wise partial finetuning pipeline called VEST and evaluate its in-distribution and out-of-distribution performance across five benchmarks.

Critical strengths
  1. The paper addresses a practical problem in few-shot recognition: checkpoint selection and hyperparameter tuning without task-specific validation data. The motivation is clear and relevant to realistic model adaptation scenarios.

  2. The proposed gF1 criterion is simple and applicable to both checkpoint and layer selection. Incorporating it into a stage-wise finetuning pipeline provides a practical approach to balancing task adaptation and generalization.

  3. The experiments report improvements in in-distribution accuracy and average out-of-distribution performance under several few-shot settings. The ablation studies and additional backbone experiments further support the effectiveness of the proposed design.

Critical weaknesses
  1. The two-step layer-selection procedure does not necessarily preserve Pareto optimality over all candidate checkpoints. Section 3.3 first retains one checkpoint per layer configuration using configuration-specific reference points, and then recomputes gF1 among the retained checkpoints. Since the reference point changes, a discarded checkpoint can dominate the final selection in both training and retrieved-data accuracy. A comparison with joint selection over all checkpoints using a common reference point would help clarify the effect of the two-step procedure.

  2. Eq. (1) computes its reference point from the minimum accuracies observed across checkpoints. Extending training can therefore change the scores and relative ranking of previously obtained checkpoints, even when the newly added checkpoints are not selected. It would be helpful to examine whether checkpoint and layer selection remain stable across different maximum training durations.

  3. The contribution of gF1 within the complete VEST pipeline is not clearly isolated. Table 3 uses gF1 in every configuration and mainly evaluates changes in the finetuning strategy. Comparing selection criteria under the same pipeline and candidate configurations would help clarify how much of the final improvement comes from the validation rule.

Questions for authors

Please see the questions raised in the weaknesses.

3 comments
AuthorsOpenReview ↗

Part 1 of our rebuttal

We appreciate the reviewer’s constructive feedback, which helped us significantly improve the quality of our manuscript; we have uploaded a revised version with all changes highlighted in blue. The reviewer recognized the strengths of our work, including our clear motivation to address a practical problem in realistic model adaptation scenarios, our simple and applicable validation method, and our comprehensive experiments and ablation studies. Below, we address each of the reviewer’s concerns and questions. Should any further clarification be needed, we would be happy to provide additional information.


The reviewer is concerned that the two-step layer-selection procedure does not necessarily preserve Pareto optimality and suggests comparing against joint selection over all checkpoints.

Thank you for this insightful question. We agree that verifying whether the two-step layer-selection procedure preserves Pareto optimality is critical. In the revised manuscript, we address this theoretically and empirically:

Theoretical guarantees. We rigorously demonstrate that the two-step procedure preserves and selects globally Pareto-optimal points

  • Theorem 1 (page 4): maximizing the harmonic mean (i.e., the F1 score) identifies a Pareto-optimal point.

  • [R1, R2]: Shifting reference baselines (e.g., minimum accuracy subtraction used in gF1) does not alter Pareto front. Consequently, our gF1 selects Pareto-optimal points.

  • Theorem 2 & Corollary 1 (page 5): The checkpoints selected by gF1, either through the two-step procedure or the joint selection paradigm, are Pareto-optimal.

Empirical comparison. In the revised manuscript, we have updated Figure 5 to compare the results of the two-step procedure and the joint selection paradigm. With both paradigms, our gF1 identifies top-4 (i.e., kk=4) layers to finetune, although selecting different checkpoints due to different reference baselines in calculating gF1 scores. Below is a synthesis of their comparison based on Figure 5 and Appendix Table 12:

approachselected kkselected ckptID test accOOD test acc
joint selection4673.5861.76
two-step selection4573.3561.80

Practical Trade-offs. We have added a discussion paragraph at Line298 analyzing the trade-offs between the two paradigms. In brief, both yield Pareto-optimal points with our gF1, although their selected layers and checkpoints can differ due to variations in the reference values used for gF1 calculation. Yet, joint selection requires retaining all checkpoints while the two-step method only needs to save one checkpoint for each kk. Therefore, the two-step paradigm offers superior storage efficiency in practice.

reference:

  • [R1]: Kaisa Miettinen et al. Nonlinear multiobjective optimization, volume 12. Kluwer Academic Publishers, Boston, 1999.
  • [R2]: R Timothy Marler and Jasbir S Arora. Survey of multi-objective optimization methods for engineering. Structural and multidisciplinary optimization, 26(6):369–395, 2004.

The reviewer asks whether gF1 remains stable for checkpoints across different maximum training durations.

In the updated manuscript, we have provided a systematic sensitivity analysis of gF1-based checkpoint selection to varying training durations (Appendix Table 11). For convenience in reviewing, we copy relevant results below, along with the result by validation-free strategy (defaulting to the last checkpoint). The results demonstrate the robustness of our gF1 to varying training durations and its consistent gains over the validation-free alternative.

epoch=20epoch=50epoch=100
methodID testOOD testavg.ID testOOD testavg.ID testOOD testavg.
validation-free72.4260.1266.2772.3259.3365.8372.5858.6165.60
gF1 score72.8361.0366.93$^{\textcolor{Green}{+0.66}}$72.5860.9466.76$^{\textcolor{Green}{+0.93}}$72.8060.8666.83$^{\textcolor{Green}{+1.23}}$
AuthorsOpenReview ↗

Part 2 of our rebuttal

The reviewer asks whether gF1 remains stable for checkpoints across different maximum training durations.

In the updated manuscript, we have provided a systematic sensitivity analysis of gF1-based checkpoint selection to varying training durations (Appendix Table 11). For convenience in reviewing, we copy relevant results below, along with the result by validation-free strategy (defaulting to the last checkpoint). The results demonstrate the robustness of our gF1 to varying training durations and its consistent gains over the validation-free alternative.

epoch=20epoch=50epoch=100
methodID testOOD testavg.ID testOOD testavg.ID testOOD testavg.
validation-free72.4260.1266.2772.3259.3365.8372.5858.6165.60
gF1 score72.8361.0366.93$^{\textcolor{Green}{+0.66}}$72.5860.9466.76$^{\textcolor{Green}{+0.93}}$72.8060.8666.83$^{\textcolor{Green}{+1.23}}$

The reviewer asks whether gF1 remains stable for layer selection across different reference points (e.g., perturbing the minimum accuracies) for gF1 calculation.

Changing the minimum reference values shifts the gF1 level sets and can alter which checkpoint is selected; yet, the underlying Pareto front of candidate checkpoints remains unchanged. We have conducted a sensitivity analysis w.r.t the ID/OOD performance of gF1-selected checkpoints with varying reference values. Specifically, using joint checkpoint and layer selection, we perturb the minimum reference values within 5 accuracy points, and re-run gF1-based selection. The results demonstrate that gF1 is highly robust to these perturbations and achieves remarkable performance gains compared with the previous strong method of full-finetuning (Liu, et al., 2025). We have added this analysis as Figure 6 in the updated manuscript, with complete numerical results in Appendix Table 12. We invite the reviewer to refer to them for visualization and in-depth analysis.


The reviewer requests a study of using different validation methods in Table 3 to validate the final improvement of applying gF1 in the VEST pipeline.

In the revised manuscript, we augmented Table 3 by adding Table 4, which compares gF1 against the validation-free strategy that defaults to selecting the last checkpoint. As the validation-free strategy lacks an inherent mechanism to determine the layer depth for finetuning, we adopt full finetuning in this comparison. For the reviewer's convenience, we reproduce Table 4 below. The results clearly isolate the contribution of gF1 and demonstrate the consistent gains of using gF1 across both training stages.

Stage-1Stage-2accuracy
tuningRAtuningID testOOD test
F××72.3259.33
F××72.58$^{\textcolor{Green}{+0.26}}$60.94$^{\textcolor{Green}{+1.61}}$
F√×71.4359.51
F√×72.89$^{\textcolor{Green}{+1.46}}$63.28$^{\textcolor{Green}{+3.77}}$
F√F73.5259.92
F√F74.49$^{\textcolor{Green}{+0.97}}$63.58$^{\textcolor{Green}{+3.66}}$

Thank you for the detailed responses. The authors have addressed all my concerns regarding the Pareto optimality of the two-step layer-selection procedure, the stability of checkpoint selection across different training durations, and the contribution of gF1 within the VEST pipeline. I appreciate the authors' efforts in addressing these points and maintain my positive evaluation of the paper.

Reviewer_mb5KOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies model selection for few-shot recognition when no additional labeled in-distribution validation set is available. The authors propose using externally retrieved data together with the few-shot training data to select checkpoints and adaptation configurations. Their gF1 metric computes the harmonic mean of training-performance gain and retrieved-data performance preservation, measured relative to the minimum accuracies among candidate checkpoints. The metric supports checkpoint and layer selection within the proposed Validation-Enabled Stage-wise Tuning (VEST) pipeline. Experiments cover five benchmarks, different shot counts, and both CLIP and DINOv2 backbones. The results demonstrate clear improvements on the main ImageNet benchmark, while the additional datasets reveal task-dependent trade-offs between ID and OOD performance.

Critical strengths

(1) Important problem and a practical validation strategy: The paper addresses a meaningful challenge in few-shot adaptation: limited labeled data constrain not only model training but also checkpoint selection and hyperparameter tuning. Using retrieved data as an additional validation signal is a useful direction, and combining this signal with training performance offers a simple approach to balancing adaptation and knowledge preservation. Figure 4 and Table 8 provide evidence that gF1 can improve checkpoint selection and can be applied to several adaptation methods. This potential applicability beyond VEST is an important strength of the contribution. (2) Strong results on the main benchmark and substantial experimental coverage: The ImageNet results are supported by three different 16-shot training splits in Table 10. VEST achieves 75.43% ID accuracy and 64.26% average OOD accuracy, compared with 73.56% and 63.37% for SWAT. Using the same retrieved training data makes this comparison particularly informative. The paper also includes experiments with another backbone, additional datasets, and analyses of layer selection, training stages, and retrieval quality and quantity. These results establish practical value and provide a substantial empirical foundation for the proposed approach.

Critical weaknesses

(1) The practical sensitivity of the selection rule to its reference values remains insufficiently examined. In Eq. (1), the two reference values are the minimum accuracies among candidate checkpoints. Adding checkpoints that change either minimum can therefore change the relative scores of existing candidates, even if the added checkpoints are not selected. Table 1 supports minimum subtraction in the reported setting, but does not assess this candidate-set dependence. Similarly, Section 3.3 first selects a checkpoint separately for each layer configuration and then recomputes gF1 across the selected candidates; the practical effect of this preliminary filtering is not directly evaluated. These observations do not establish an error in the method or the reported results. However, because the main contribution is a model-selection strategy, the practical consequences of these choices are important to its assessment. An analysis that extends or truncates the candidate range on fixed training trajectories would help establish whether changes in the reference values materially affect the selected models and their performance. Comparing the two-step layer-selection procedure with joint selection under a shared reference point would also help evaluate this design choice. (2) The evidence for the additional benefit of the selector within VEST could be more direct. Figure 4 and Table 8 provide useful evidence for gF1, and Table 3 examines the effects of retrieval augmentation, partial finetuning, and stage-wise training while using gF1 throughout. However, Table 3 does not compare alternative selection rules within the same VEST configuration. A comparison with retrieved-data accuracy alone and a prespecified checkpoint policy, while holding the training configuration and candidate checkpoints fixed, would clarify the additional benefit of gF1. This comparison is particularly relevant given Figure 10. For DINOv2 checkpoint selection, gF1 improves ID accuracy over selection by retrieved-data accuracy alone from 77.32% to 77.73%, but decreases average OOD accuracy from 59.89% to 58.50%. This result demonstrates a trade-off rather than a joint improvement over that simple alternative. Although gF1 improves over the final checkpoint, the mixed comparison with retrieved-data-based selection warrants a more precise account of when the proposed rule is preferable.

Questions for authors

(1) On fixed, already-trained trajectories, how much do the selected models and their ID/OOD performance change when extending the candidate range changes the minima in Eq. (1)? How does the two-step layer-selection procedure compare with joint selection over all layer–checkpoint candidates using a shared reference point? (2) For a fixed VEST layer configuration and Stage-1 model, how does gF1 compare with retrieved-data accuracy alone and a prespecified checkpoint policy on the same Stage-2 training trajectory? This comparison would help isolate its contribution to final-stage checkpoint selection. (3) Using the overall-average convention reported in Tables 8–9, the retrieved-data selector in Figure 10 obtains approximately 68.61%, compared with 68.12% for gF1. How do the authors interpret this result, and what evidence indicates the settings in which gF1 offers a preferable ID/OOD balance?

3 comments
AuthorsOpenReview ↗

Part 1 of our rebuttal

We appreciate the reviewer’s constructive feedback, which helped us significantly improve the quality of our manuscript; we have uploaded a revised version with all changes highlighted in blue. The reviewer recognized the strengths of our work, including our practical validation strategy, the importance of our formulated problem, and our strong results and practical value. Below, we address each of the reviewer’s concerns and questions. Should any further clarification be needed, we would be happy to provide additional information.


The reviewer asks how changes in minimum reference values affect the ID/OOD performance of gF1 on fixed trajectories.

Changing the minimum reference values shifts the gF1 level sets and can alter which checkpoint is selected; yet, the underlying Pareto front of candidate checkpoints remains unchanged. We have conducted a sensitivity analysis w.r.t the ID/OOD performance of gF1-selected checkpoints with varying reference values. Specifically, using joint checkpoint and layer selection, we perturb the minimum reference values within 5 accuracy points, and re-run gF1-based selection. The results demonstrate that gF1 is highly robust to these perturbations and achieves remarkable performance gains compared with the previous strong method of full-finetuning (Liu, et al., 2025). We have added this analysis as Figure 6 in the updated manuscript, with complete numerical results in Appendix Table 12. We invite the reviewer to refer to them for visualization and in-depth analysis.


The reviewer asks for a comparison against the joint selection using our gF1 score in the two-step layer selection procedure.

Great question! In the revised manuscript, we have updated Figure 5 to show the results of joint layer and checkpoint selection. Using our gF1 score, joint selection determines kk=4, i.e., finetuning the top-4 layers. This matches the two-step procedure, although selecting a different checkpoint due to different reference baselines in calculating gF1 scores. Below is a synthesis of their comparison based on Figure 5 and Appendix Table 12:

approachselected kkselected ckptID test accOOD test acc
joint selection4673.5861.76
two-step selection4573.3561.80

Furthermore, we have added a discussion paragraph at Line295 analyzing the trade-offs between the joint and two-step paradigms. In brief, both methods yield Pareto-optimal points, although their selected layers and checkpoints can differ due to variations in the reference values used for gF1 calculation. Yet, joint selection requires retaining all checkpoints while the two-step method only needs to save one checkpoint for each kk. Therefore, the two-step paradigm offers superior storage efficiency in practice.

Lately, we highlight that the revised manuscript now features Theorem 1 (page 4) along with its formal proof in Appendix A. This theorem formally establishes that maximizing the F1 score guarantees the selection of a Pareto-optimal point. Moreover, as noted on Line 233, shifting the reference values, i.e., minimum accuracy as the reference baselines, preserves the underlying Pareto front and focuses the actual trade-off boundary. These theoretically demonstrate the effectiveness and robustness of our gF1 score.

AuthorsOpenReview ↗

Part 2 of our rebuttal

The reviewer requests a comparison of different checkpoint selection methods at stage-2 of VEST, under a fixed training and layer configuration (i.e., using the same Stage-1 model, cf. Table 3). The methods should include the proposed gF1, retrieved-data accuracy, and "a prespecified checkpoint policy".

To effectively address this question, let us exemplify "a prespecified checkpoint policy" as the validation-free strategy, which defaults to the last checkpoint and is widely adopted in the literature (Liu et al., 2025; Silva-Rodriguez et al., 2024). The table below compares the results of stage-2 checkpoints selected by different validation methods. Here, they run with the same stage-1 model, which is the last checkpoint via FFT w/ RA. The results confirm the superior performance of our gF1.

validation methodID testOOD test
validation-free73.5259.92
retrieved data73.1861.71
gF1 score73.7861.60

This comparison further motivates an end-to-end evaluation in which the same validation method is applied consistently across both training stages. Concretely, to ensure a fair comparison and isolate the effectiveness of each checkpoint selection approach, all methods employ full finetuning, as the validation-free strategy lacks an inherent mechanism for layer selection. The results below demonstrate that gF1 consistently outperforms the alternatives in this end-to-end evaluation.

val methodStage-1Stage-2accuracy
tuningRAtuningID testOOD test
val-freeF√F73.5259.92
retrieved data accuracyF√F73.1562.93
gF1F√F74.4963.58

The reviewer notes that, when finetuning DINOv2 (cf. Appendix Figure 11 in the updated manuscript), using the accuracy on retrieved data performs as well as the proposed gF1, with the two exhibiting trade-offs on the ID and OOD test accuracy. The reviewer asks for an explanation.

Indeed, with DINOv2 on the ImageNet-OOD benchmark, selecting checkpoints via retrieved-data accuracy performs comparably to our gF1 (exhibiting a minor ID/OOD trade-off), where gF1 substantially outperforms retrieved-data accuracy when using CLIP (ref. Fig. 4). We attribute this to two key factors:

  • Stronger feature alignment in DINOv2. DINOv2 learns more generalizable features, as evidenced by its higher ID test accuracy compared with CLIP (Table 2). Consequently, in the DINOv2 feature space, the retrieved data are substantially closer to the ID distribution (i.e., less OOD). As the distribution gap between ID and retrieved data is smaller, raw accuracy on the retrieved data serves well as a validation proxy on the ImageNet-OOD benchmark.

  • Context of fair baselines. We note that leveraging retrieved data as a validation set itself is a contribution of our work, as prior literature primarily relies on a validation-free strategy (defaulting to the last checkpoint). Compared against the prior standard validation-free baseline, our gF1 consistently yields significant gains across all backbones.

Thank you for the additional experiments. The comparison using the same Stage‑1 model addresses my main concern about the contribution of checkpoint selection. In that setting, gF1 improves both ID and OOD accuracy over the last checkpoint, although selecting by retrieved-data accuracy gives a slightly higher OOD result. When the same selection rule is applied in both stages of full fine-tuning, gF1 gives the highest ID and OOD accuracy among the three methods compared.The comparison between joint and two-step selection was also helpful: both chose the top four layers, and the different checkpoints they selected gave similar test results. Based on this additional evidence, I am raising my rating from 2 (Weak rejection) to 3 (Weak acceptance).

Reviewer_hHX7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies model selection and hyperparameter tuning for few-shot recognition (FSR) under adaptation of pretrained vision-language models (VLM), aiming for robust generalization to out-of-distribution (OOD) test data. In the realistic setting, no task-specific validation set is available. The authors start from two observations. First, fine-tuning can distort pretrained features and hurt generalization. Second, images retrieved from external sources can serve as validation data, but validating directly on retrieval accuracy is self-defeating: as the model keeps fitting the few-shot in-distribution (ID) training data, its accuracy on the retrieved data keeps dropping, so conventional validation favors the untuned model or a very early checkpoint. To resolve this dilemma, the authors formulate model selection as a trade-off between performance gain on the ID training data and performance preservation on the retrieved data, and propose the gF1 score: it subtracts the minimum accuracy within the candidate checkpoint pool and then takes the harmonic mean of the two resulting increments. The metric supports both checkpoint selection and the selection of layers for partial fine-tuning (PFT), and is integrated into a three-stage pipeline, VEST. In Stage 0, gF1 selects the PFT layers; in Stage 1, the selected layers are fine-tuned on the mixture of ID and retrieved data; in Stage 2, they are further fine-tuned on ID data only. On the ImageNet-based OOD benchmark under 4/8/16-shot settings, VEST achieves the highest ID accuracy and average OOD accuracy among all compared methods; experiments on Living17, Entity30, fMoW, and DomainNet further examine its applicability, although the full VEST is not always better than its PFT variant. Supplementary experiments show that gF1 is robust to changes in the retrieved data, transfers to four adaptation families, and works with DINOv2 backbones.

Critical strengths

The main strength of this paper is that it identifies and systematically studies the model selection problem under the absence of a task-specific validation set in realistic few-shot adaptation. Through extensive experiments, the authors demonstrate the limits of relying on training accuracy or retrieved accuracy alone, and propose gF1, a simple method applicable to both checkpoint and hyperparameter selection. Component ablations, baseline comparisons under the same retrieved data, and extensions across adaptation methods and backbones comprehensively examine the practical value of the strategy; notably, even when baselines are allowed to tune on the ID test set, a setting in their favor, VEST still achieves leading ID and OOD accuracy.Finally, this method also has scalability and can be applied in other methods or models.

Critical weaknesses

The authors provide no argument or analysis showing that the checkpoint selected by gF1 is truly the one with the best generalization. It is essentially a heuristic score assembled from two accuracies, and the Pareto terminology is only packaging: the paper does not prove that the selected point is Pareto optimal in any sense. What if a practitioner changes the combination of these components, and how should the optimal combination be chosen? More importantly, the causal evidence is missing. Every ablation in Table 3 uses gF1 for checkpoint selection, and a version of the stage-wise pipeline without validation has never been run. Decomposing the numbers in Table 3, most of the 16-shot ID improvement over SWAT is explained by replacing SWAT's classifier-only stage 2 with full fine-tuning (+1.01; this row still uses gF1 for selection, so the pure recipe contribution may be slightly lower), while the visible contribution of gF1's layer selection across both stages is modest (about +0.66 ID in total, of which only +0.01 ID and +0.17 OOD come from stage 2). No row in the table tells us how much of the gain would remain if neither stage used validation. The min-subtraction turns gF1 into a relative score that depends on the candidate pool. When the pool size, training length, or sampling density changes, the same checkpoint receives a different score. The paper provides no systematic sensitivity analysis, e.g., whether the selected checkpoint changes under different training lengths, sampling intervals, or retrieval budgets. Table 9 on DINOv2 already hints at this: gF1 fluctuates between 0.00 and 0.43 as k varies, with several configurations scoring exactly 0, which suggests that the metric may not be stable.

Questions for authors

Could you add a no-validation control to Table 3 (last checkpoint, default hyperparameters, fixed top-k layers) and report the marginal contribution of gF1? The top-500 most similar retrieved images per class are reserved for training, and the validation set is randomly sampled from the remainder, so it consists systematically of lower-similarity images. Why is it designed this way? If higher-similarity images were used for validation instead, would the paradox in Fig. 3 still be as pronounced? The paper uses the gF1 score to represent generalization, but neither the main text nor the appendix explains why this score connects to OOD test performance. Could the authors provide an analysis, even under explicit assumptions?

2 comments
AuthorsOpenReview ↗

Part 1 of our rebuttal

We appreciate the reviewer’s constructive feedback, which helped us significantly improve the quality of our manuscript; we have uploaded a revised version with all changes highlighted in blue. The reviewer recognized the strengths of our work, including our systematic study, our simple and applicable method, and the practical value of our validation strategy. Below, we address each of the reviewer’s concerns and questions. Should any further clarification be needed, we would be happy to provide additional information.


There is no argument showing that the gF1-selected checkpoint is the one with the best generalization. The paper does not prove that the gF1-selected checkpoint is Pareto optimal.

Great question! In the revised manuscript, we added Theorem 1 on page 4 along with its formal proof in Appendix A, establishing that maximizing the harmonic mean (the F1 score) is guaranteed to select a Pareto-optimal checkpoint. Furthermore, shifting the reference baselines via minimum-accuracy subtraction preserves the Pareto front while restoring the harmonic mean’s sensitivity to imbalanced trade-offs, a crucial property when individual metrics exhibit high baseline values, such as training accuracy [R1, R2]. We invite the reviewer to examine these additions in the updated manuscript, and we sincerely thank the reviewer for stimulating us to strengthen the theoretical foundation of our work.

reference:

  • [R1] Peter Flach and Meelis Kull. Precision-recall-gain curves: Pr analysis done right. Advances in neural information processing systems, 28, 2015.
  • [R2] David MW Powers. Evaluation: from precision, recall and f-measure to roc, informedness, markedness and correlation. arXiv preprint arXiv:2010.16061, 2020.

The reviewer suggests adding "a no-validation control to Table 3 (last checkpoint, default hyperparameters, fixed top-k layers) and report the marginal contribution of gF1".

In the revised manuscript, we augmented Table 3 by adding Table 4, which compares gF1 against the validation-free strategy that defaults to selecting the last checkpoint. As the validation-free strategy lacks an inherent mechanism to determine the layer depth for finetuning, we adopt full finetuning in this comparison. For the reviewer's convenience, we reproduce Table 4 below. The results clearly isolate the contribution of gF1 and demonstrate the consistent gains of gF1 across both training stages.

val methodStage-1Stage-2accuracy
tuningRAtuningID testOOD test
val-freeF××72.3259.33
gF1F××72.58$^{\textcolor{Green}{+0.26}}$60.94$^{\textcolor{Green}{+1.61}}$
val-freeF√×71.4359.51
gF1F√×72.89$^{\textcolor{Green}{+1.46}}$63.28$^{\textcolor{Green}{+3.77}}$
val-freeF√F73.5259.92
gF1F√F74.49$^{\textcolor{Green}{+0.97}}$63.58$^{\textcolor{Green}{+3.66}}$
AuthorsOpenReview ↗

Part 2 of our rebuttal

The reviewer asks (1) why using the lower-similarity images for validation, and (2) if using higher similarity images for validation would change the paradox in Fig. 3.

We did not intentionally select lower-similarity retrieved images for validation; rather, we used exactly the same set of retrieved images as the previous method, SWAT (Liu et al., 2025), to ensure a fair comparison. As a result, we used the remaining retrieved images for validation. To address the reviewer's concern, we added Fig. 13 in the Appendix of the revised manuscript, which reports checkpoint accuracy evaluated on the high-similarity retrieved images, i.e., the top-50 retrieved samples with the highest feature similarity to the ID training samples (i.e., the same samples referred to as "high-quality" in Table 5). The results confirm that the paradox observed in Fig. 3 still holds.


The reviewer asks for an analysis of why the gF1 score connects to OOD test performance.

Recall that the gF1 score is designed to balance the performance gains on ID training data and performance preservation on retrieved data. Because the retrieved data originates from outside the ID training distribution, albeit with label noise (Fig. 2), it naturally serves as a proxy "noisy" OOD validation set. More specifically, the empirical and geometric connection between gF1 and OOD test performance is grounded in two key observations:

  • Correlated optimization dynamics: As shown in Fig. 3, accuracy trajectories on the retrieved data and the OOD test data exhibit closely aligned trends throughout finetuning. This strong correlation indicates that performance on the retrieved data provides a signal for tracking OOD test performance.

  • Feature space geometry: As shown in the feature visualizations in Appendix Fig. 9, both the OOD test data and the retrieved data shift away from the ID training data, with many retrieved samples lying in the intermediate region between the ID and OOD data points. Geometrically, preserving performance on these intermediate samples constrains the model from overfitting solely to the ID data, thereby bridging the gap toward OOD test distributions

In sum, by exploiting accuracy on the retrieved data alongside the ID training accuracy, the gF1 score effectively reflects OOD generalization and serves as a reliable validation method.


The reviewer requests systematic analysis of the gF1 score to varying training lengths, sampling intervals, and retrieval budgets.

We have conducted a systematic sensitivity analysis of the gF1 score regarding varying training lengths, checkpoint sampling intervals, and retrieval budget (i.e., the number of retrieved images). These results are incorporated into the updated manuscript as Appendix Table 11, Appendix Fig. 12, and Table 5, respectively, and are reproduced below for convenience. For each table, we also include a validation-free baseline (defaulting to the last checkpoint without checkpoint selection) as a reference. Across all variations, the results demonstrate that gF1 is highly robust to these changes and consistently outperforms the validation-free alternative.

  • gF1 is robust to varying training lengths (Appendix Table 11): | | epoch=20 | | | epoch=50 | | | epoch=100 | | | | - | :- | :- | :- | :- | :- | :- | :- | - | :- | | validation method | ID test | OOD test | avg. | ID test | OOD test | avg. | ID test | OOD test | avg. | | validation-free|72.42| 60.12| 66.27 | 72.32| 59.33 | 65.83| 72.58 | 58.61 | 65.60 | | gF1 score| 72.83 | 61.03 | 66.93$^{\textcolor{Green}{+0.66}}$ | 72.58 | 60.94 | 66.76$^{\textcolor{Green}{+0.93}}$ | 72.80 | 60.86 | 66.83$^{\textcolor{Green}{+1.23}}$ |

  • gF1 is robust to varying sampling intervals (Appendix Fig. 12): | sampling interval | ID test | OOD test | avg. | | - | :-: | :-: | :-: | | validation-free | 72.32 | 59.33 | 65.83 | | 100 iters | 72.41 | 61.05 | 66.73 | | 150 iters | 72.55 | 61.03 | 66.79 | | 200 iters | 72.74 | 60.88 | 66.81 | | 250 iters | 72.58 | 60.94 | 66.76 |

  • gF1 is robust to varying retrieval budgets (Table 5): | | ckpt | ID test set | OOD test set | | - | :-: | :-: | :-: | | validation-free | 50 | 72.32 | 59.33 | | rand-25| 4 | 72.75 | 60.82 | | rand-50 | 3 | 72.58 | 60.94 | | rand-100 | 3 | 72.58 | 60.94 |


The reviewer thinks that our gF1 is not stable based on the results in Appendix Table 9, which shows zero gF1 scores for some entries.

Table 9 studies using gF1 for layer selection with the DINOv2 backbone. The zero gF1 scores do not indicate instability; rather, they simply mean that the gF1-select checkpoint for layer kk does not significantly outperform the reference baseline (cf. L303 and Fig.5 in updated manuscript). In fact, our gF1 selects the top-1 layer of DINOv2 to finetune, which consistently outperforms other baselines such as linear probing and full finetuning (Table 9). This actually confirms the robustness of gF1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.