- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
GAFT improves semi-supervised, few-shot adaptation by treating each labeled example as an anchor. Unlabeled examples are assigned to their nearest anchor in backbone embedding space, with assignments refreshed during fine-tuning. The method adds a density-weighted supervised loss, anchor-to-unlabeled prediction alignment, and within-region prediction smoothness to an existing SSL objective. A risk bound motivates these terms. Experiments cover six VTAB tasks with DINOv2 and two PEFT mechanisms, plus additional backbones and robustness settings.
Critical strengths
- The improvements are substantial across heterogeneous tasks. In Table 1, V-PET with LoRA improves from 55.85% to 62.10% average accuracy, including CLEVR-C from 36.8% to 51.84%. With AdaptFormer, its average improves from 56.27% to 60.08%. This supports practical value beyond a single dataset or adaptation mechanism.
- The appendix contains meaningful controls. Table 12 retains reweighted supervision while varying the regularizers; for PET/LoRA, accuracy improves from 67.59% without either regularizer to 70.41% with both. Table 17 reports matched LP and Laplacian comparisons, and Table 15 includes a matched GAFT removal on Retinopathy. These strengthen the evidence that the proposed addition contributes beyond simply reporting stronger final models.
Critical weaknesses
- The theory-to-algorithm link remains incomplete. Appendix A.2 conditions on a fixed partition and applies multinomial concentration to its region counts. After the backbone has been trained using the same unlabeled pool, the refreshed partition in Section 4.2 is itself sample-dependent; the conditional multinomial argument does not automatically extend to this setting. Separately, Section 4.1 replaces the population discrepancy in Eq. 4 with within-pool pairwise discrepancy in Eq. 7 without bounding that approximation for the trained predictor. The paper explicitly calls these empirical surrogates, and the fixed-partition result remains useful motivation. However, the claims of direct correspondence to the bound need a clearer scope.
- Attribution of the main gains is less controlled than the additive-objective description suggests. Section 5.1 and the reproducibility statement say the Table 1 baselines come from prior work, while GAFT also introduces cluster-conditioned batching and extra supervised loss. The stated shared recipe does not specify whether labeled draws, teacher targets, and sampling are matched. Table 12 helps isolate the regularizers on DTD, but the largest RESISC45/CLEVR-C gains remain comparisons against sourced baselines. Three-seed means without dispersion also leave uncertainty around smaller improvements.
Questions for authors
- Is Theorem 3.5 intended to cover only an independently specified partition? Which part of the periodically refreshed training procedure is covered by the current probability statement?
- What approximation control or additional assumptions are intended when replacing the population discrepancy by Eq. 7 for a predictor trained on that same pool?
- Which baseline/+GAFT comparisons share labeled draws, teacher targets, and batching? Please clarify this and summarize variability available from the existing runs, particularly for the largest gains.
2 comments
Addressing W2 & Q3
We would like to continue addressing the reviewer's concerns in this comment.
W2: "Attribution of the main gains is less controlled than the additive-objective description suggests... The stated shared recipe does not specify whether labeled draws, teacher targets, and sampling are matched. Table 12 helps isolate the regularizers on DTD, but the largest RESISC45/CLEVR-C gains remain comparisons against sourced baselines. Three-seed means without dispersion also leave uncertainty around smaller improvements."
Q3: "Which baseline/+GAFT comparisons share labeled draws, teacher targets, and batching? Please clarify this and summarize variability available from the existing runs, particularly for the largest gains."
About the shared recipe in configuring experiments
All runs use the released labeled splits of Zhang et al. (2025), and the three seeds only change the mini-batch sampling, not the split. The pseudo-labels (the reviewer's teacher targets) are obtained in a step prior to fine-tuning, by ensembling the predictions of PEFT models fine-tuned on the limited labeled data only, using the released code. Hence a run with and without GAFT trains on one and the same set of pseudo-labels, and the base SSL objective is left unmodified (Eq. 10). In other words, labeled draws, teacher targets, and batching are all shared across baseline/+GAFT comparisons and across seeds.
While the main results table source from previously reported results, to cement our work, we have also rerun PET/V-PET under the same splits, seeds, preprocessing and training budget (reported in our response to Reviewer TchU).
About attribution of the gains
Combining these reruns with Table 12, where only and vary, separates the gain on DTD (3-shot) into its parts:
| DTD (3) | Rerun baseline | GAFT with | Full GAFT |
|---|---|---|---|
| LoRA, PET | 66.17 | 67.59 | 70.41 |
| LoRA, V-PET | 67.25 | 68.03 | 69.17 |
| AdaptFormer, PET | 67.14 | 67.47 | 69.34 |
| AdaptFormer, V-PET | 68.21 | 68.29 | 69.13 |
In all four configurations, the two regularizers account for the majority of the total gain, while GAFT with both regularizers disabled is +0.08 to +1.42 above the baseline.
About variability for existing runs and results
We thank the reviewer for the suggestion. To elaborate on the results, we report the mean ± standard deviation for additional datasets, over the three seeds, as follows.
| PEFT | Method | DTD (3) | KITTI (5) | RESISC45 (1) | CLEVR-C (10) |
|---|---|---|---|---|---|
| LoRA | PET (rerun) | 66.17 ± 0.38 | 43.95 ± 0.59 | 55.28 ± 0.17 | 33.08 ± 0.12 |
| PET + GAFT | 70.41 ± 0.24 | 55.73 ± 0.41 | 62.86 ± 0.23 | 32.05 ± 0.20 | |
| V-PET (rerun) | 67.25 ± 0.30 | 58.6 ± 0.53 | 65.99 ± 0.45 | 36.80 ± 0.75 | |
| V-PET + GAFT | 69.17 ± 0.27 | 61.67 ± 0.33 | 78.95 ± 0.74 | 51.84 ± 0.98 | |
| AdaptFormer | PET (rerun) | 67.14 ± 0.38 | 45.85 ± 0.64 | 56.24 ± 0.55 | 33.36 ± 0.54 |
| PET + GAFT | 69.34 ± 0.26 | 58.68 ± 0.30 | 62.85 ± 0.32 | 37.40 ± 0.41 | |
| V-PET (rerun) | 68.21 ± 0.19 | 59.02 ± 0.29 | 66.24 ± 0.27 | 36.89 ± 0.72 | |
| V-PET + GAFT | 69.13 ± 0.27 | 58.90 ± 0.51 | 75.61 ± 0.46 | 48.14 ± 0.59 |
On KITTI, where PET + GAFT yields large gains in Table 1, the improvements over our reruns (+11.78 with LoRA, +12.83 with AdaptFormer) exceed the seed standard deviations by more than an order of magnitude. A similar pattern can be observed for runs that show substantial improvements, such as for RESISC45 and CLEVR-C. Overall, almost every other gain in the table also exceeds the sum of the two standard deviations, showing us the empirical benefits of GAFT in fine-tuning for such tasks.
Once again, we thank the reviewer for taking the time to assess our work and acknowledging GAFT's strengths. We hope these comments have fully addressed your concerns.
Addressing W1, Q1, & Q2
We would like to thank the reviewer for carefully reading the theory and for acknowledging both the substantial improvements of GAFT across tasks and the meaningful controls in our appendix. We would like to address your remaining concerns as follows.
To address these points, in our updated PDF, we have added a number of theoretical results: Theorem 3.7 (Section 3.4, proof in Appendix A.4) for the refreshed partition, and Theorem A.5 (Appendix A.7) for the empirical surrogate. Equation numbers below follow the updated PDF, where the reviewer's Eq. 7 is now Eq. 8.
About how the theoretical analyses cover the actual training procedure
Theorem 3.5 covers only a partition that is independent of Du. In GAFT, this is exactly the initial partition: following PET/V-PET, fine-tuning starts from the pretrained VFM, and Γ is obtained by assigning each point to its nearest labeled anchor in this pretrained embedding space (Section 4.2.1), so it depends on D but not on Du.
The refreshed partitions (Section 4.2.2) are indeed sample-dependent, and this is what Theorem 3.7 addresses. It holds for any partition that depends on D or Du, by decoupling concentration from adaptation: the multinomial argument is applied only to a reference partition Γ∗ that is independent of Du, and the change introduced by the data-dependent partition is paid for explicitly by the term C TV(Q∗,Q). Without refreshing (Γ=Γ∗), this term vanishes and Theorem 3.5 is recovered. Finally, Theorem A.5 holds simultaneously for all h∈H and all partitions in an admissible family G, so it also covers a predictor trained on the same pool.
About the empirical surrogate of the population term
Theorem A.5 replaces the population term d(Diu,Xi∣h) of Theorem 3.7 with exactly the within-pool discrepancy Rrobust used in training. The additional assumptions are (i) bounded prediction distances, ∥h(a)−h(x)∥≤B for all h∈H, (ii) a minimum mass p∗>0 for every admissible region, and (iii) a family G of admissible partitions fixed independently of Du, while the selected partition may depend on Du. Under these, the gap between the population local variation and Rrobust is controlled explicitly by empirical Rademacher complexities (Bartlett & Mendelson, 2002) scaled by 1/p∗ (Eq. 32). Since this holds uniformly over H, it applies to the predictor trained on that same pool, which is precisely the reviewer's concern.
Substituting this into Theorem 3.7 gives the full bound (Eq. 28), whose h-dependent part is ∑iπ^iℓ(h,si)+LRalign(h)+LRrobust(h), i.e. LGAFT with λ1=λ2=L. To state the scope as clearly as the reviewer asks: minimizing LGAFT directly optimizes these three empirical h-dependent terms, while the partition-discrepancy, label (gu) and complexity terms are explicit in the bound but not optimized by it (Appendix A.7), and we do not claim that they are numerically small. The correspondence is thus between LGAFT and the h-dependent part of a valid bound for the actual refreshed procedure.
Once again, we would like to thank the reviewer for raising these points. We will address subsequent points in the next Official Comment.