acceptodds
Under review as a conference paper at ICLR 2027

GAFT: Geometry-Aware Fine-Tuning for Foundation Models

Abstract

Adapting Vision Foundation Models (VFMs) in low-label regimes remains challenging despite their strong transferable representations. While semi-supervised learning (SSL) can mitigate label scarcity, most existing approaches treat unlabeled data primarily as noisy supervision, without explicitly accounting for the geometry of the learned feature space. In this work, we develop a theoretical framework showing that unlabeled data improves generalization by revealing the structure and density of the representation space, yielding a model-adaptive and data-adaptive bound in which labeled samples act as geometric anchors and local prediction robustness governs expected risk. We then propose GAFT (Geometry-Aware Fine-Tuning) which introduces geometry-aware regularization terms that integrate seamlessly into existing SSL pipelines. Extensive experiments on the Visual Task Adaptation Benchmark (VTAB), across multiple fine-tuning strategies, demonstrate that GAFT consistently improves both accuracy and robustness over strong baselines, highlighting the effectiveness of geometry-aware SSL for VFMs.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Semi-supervised Learning, Few-shot Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MNy9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GAFT improves semi-supervised, few-shot adaptation by treating each labeled example as an anchor. Unlabeled examples are assigned to their nearest anchor in backbone embedding space, with assignments refreshed during fine-tuning. The method adds a density-weighted supervised loss, anchor-to-unlabeled prediction alignment, and within-region prediction smoothness to an existing SSL objective. A risk bound motivates these terms. Experiments cover six VTAB tasks with DINOv2 and two PEFT mechanisms, plus additional backbones and robustness settings.

Critical strengths
  1. The improvements are substantial across heterogeneous tasks. In Table 1, V-PET with LoRA improves from 55.85% to 62.10% average accuracy, including CLEVR-C from 36.8% to 51.84%. With AdaptFormer, its average improves from 56.27% to 60.08%. This supports practical value beyond a single dataset or adaptation mechanism.
  2. The appendix contains meaningful controls. Table 12 retains reweighted supervision while varying the regularizers; for PET/LoRA, accuracy improves from 67.59% without either regularizer to 70.41% with both. Table 17 reports matched LP and Laplacian comparisons, and Table 15 includes a matched GAFT removal on Retinopathy. These strengthen the evidence that the proposed addition contributes beyond simply reporting stronger final models.
Critical weaknesses
  1. The theory-to-algorithm link remains incomplete. Appendix A.2 conditions on a fixed partition and applies multinomial concentration to its region counts. After the backbone has been trained using the same unlabeled pool, the refreshed partition in Section 4.2 is itself sample-dependent; the conditional multinomial argument does not automatically extend to this setting. Separately, Section 4.1 replaces the population discrepancy in Eq. 4 with within-pool pairwise discrepancy in Eq. 7 without bounding that approximation for the trained predictor. The paper explicitly calls these empirical surrogates, and the fixed-partition result remains useful motivation. However, the claims of direct correspondence to the bound need a clearer scope.
  2. Attribution of the main gains is less controlled than the additive-objective description suggests. Section 5.1 and the reproducibility statement say the Table 1 baselines come from prior work, while GAFT also introduces cluster-conditioned batching and extra supervised loss. The stated shared recipe does not specify whether labeled draws, teacher targets, and sampling are matched. Table 12 helps isolate the regularizers on DTD, but the largest RESISC45/CLEVR-C gains remain comparisons against sourced baselines. Three-seed means without dispersion also leave uncertainty around smaller improvements.
Questions for authors
  1. Is Theorem 3.5 intended to cover only an independently specified partition? Which part of the periodically refreshed training procedure is covered by the current probability statement?
  2. What approximation control or additional assumptions are intended when replacing the population discrepancy by Eq. 7 for a predictor trained on that same pool?
  3. Which baseline/+GAFT comparisons share labeled draws, teacher targets, and batching? Please clarify this and summarize variability available from the existing runs, particularly for the largest gains.
2 comments
AuthorsOpenReview ↗

Addressing W1, Q1, & Q2

We would like to thank the reviewer for carefully reading the theory and for acknowledging both the substantial improvements of GAFT across tasks and the meaningful controls in our appendix. We would like to address your remaining concerns as follows.


W1: "The theory-to-algorithm link remains incomplete... the refreshed partition in Section 4.2 is itself sample-dependent; the conditional multinomial argument does not automatically extend to this setting. Separately, Section 4.1 replaces the population discrepancy in Eq. 4 with within-pool pairwise discrepancy in Eq. 7 without bounding that approximation for the trained predictor..."

Q1: "Is Theorem 3.5 intended to cover only an independently specified partition? Which part of the periodically refreshed training procedure is covered by the current probability statement?"

Q2: "What approximation control or additional assumptions are intended when replacing the population discrepancy by Eq. 7 for a predictor trained on that same pool?"

To address these points, in our updated PDF, we have added a number of theoretical results: Theorem 3.7 (Section 3.4, proof in Appendix A.4) for the refreshed partition, and Theorem A.5 (Appendix A.7) for the empirical surrogate. Equation numbers below follow the updated PDF, where the reviewer's Eq. 7 is now Eq. 8.

About how the theoretical analyses cover the actual training procedure

Theorem 3.5 covers only a partition that is independent of DuD^u. In GAFT, this is exactly the initial partition: following PET/V-PET, fine-tuning starts from the pretrained VFM, and Γ\Gamma is obtained by assigning each point to its nearest labeled anchor in this pretrained embedding space (Section 4.2.1), so it depends on DD but not on DuD^u.

The refreshed partitions (Section 4.2.2) are indeed sample-dependent, and this is what Theorem 3.7 addresses. It holds for any partition that depends on DD or DuD^u, by decoupling concentration from adaptation: the multinomial argument is applied only to a reference partition Γ∗\Gamma^\ast that is independent of DuD^u, and the change introduced by the data-dependent partition is paid for explicitly by the term C TV(Q∗,Q)C\ \mathrm{TV}(Q^\ast, Q). Without refreshing (Γ=Γ∗\Gamma = \Gamma^\ast), this term vanishes and Theorem 3.5 is recovered. Finally, Theorem A.5 holds simultaneously for all h∈Hh \in \mathcal{H} and all partitions in an admissible family G\mathcal{G}, so it also covers a predictor trained on the same pool.

About the empirical surrogate of the population term

Theorem A.5 replaces the population term d(Diu,Xi∣h)d(D^u_i,\mathcal X_i\mid h) of Theorem 3.7 with exactly the within-pool discrepancy Rrobust\mathcal R_{\rm robust} used in training. The additional assumptions are (i) bounded prediction distances, ∥h(a)−h(x)∥≤B\Vert h(a)-h(x)\Vert\le B for all h∈Hh\in\mathcal H, (ii) a minimum mass p∗>0p^\ast>0 for every admissible region, and (iii) a family G\mathcal G of admissible partitions fixed independently of DuD^u, while the selected partition may depend on DuD^u. Under these, the gap between the population local variation and Rrobust\mathcal R_{\rm robust} is controlled explicitly by empirical Rademacher complexities (Bartlett & Mendelson, 2002) scaled by 1/p∗1/p^\ast (Eq. 32). Since this holds uniformly over H\mathcal H, it applies to the predictor trained on that same pool, which is precisely the reviewer's concern.

Substituting this into Theorem 3.7 gives the full bound (Eq. 28), whose hh-dependent part is ∑iπ^iℓ(h,si)+L Ralign(h)+L Rrobust(h)\sum_i\hat\pi_i\ell(h,s_i)+L\,\mathcal R_{\rm align}(h)+L\,\mathcal R_{\rm robust}(h), i.e. LGAFT\mathcal L_{\rm GAFT} with λ1=λ2=L\lambda_1=\lambda_2=L. To state the scope as clearly as the reviewer asks: minimizing LGAFT\mathcal L_{\rm GAFT} directly optimizes these three empirical hh-dependent terms, while the partition-discrepancy, label (gug_u) and complexity terms are explicit in the bound but not optimized by it (Appendix A.7), and we do not claim that they are numerically small. The correspondence is thus between LGAFT\mathcal L_{\rm GAFT} and the hh-dependent part of a valid bound for the actual refreshed procedure.

Once again, we would like to thank the reviewer for raising these points. We will address subsequent points in the next Official Comment.

AuthorsOpenReview ↗

Addressing W2 & Q3

We would like to continue addressing the reviewer's concerns in this comment.


W2: "Attribution of the main gains is less controlled than the additive-objective description suggests... The stated shared recipe does not specify whether labeled draws, teacher targets, and sampling are matched. Table 12 helps isolate the regularizers on DTD, but the largest RESISC45/CLEVR-C gains remain comparisons against sourced baselines. Three-seed means without dispersion also leave uncertainty around smaller improvements."

Q3: "Which baseline/+GAFT comparisons share labeled draws, teacher targets, and batching? Please clarify this and summarize variability available from the existing runs, particularly for the largest gains."

About the shared recipe in configuring experiments

All runs use the released labeled splits of Zhang et al. (2025), and the three seeds only change the mini-batch sampling, not the split. The pseudo-labels (the reviewer's teacher targets) are obtained in a step prior to fine-tuning, by ensembling the predictions of PEFT models fine-tuned on the limited labeled data only, using the released code. Hence a run with and without GAFT trains on one and the same set of pseudo-labels, and the base SSL objective is left unmodified (Eq. 10). In other words, labeled draws, teacher targets, and batching are all shared across baseline/+GAFT comparisons and across seeds.

While the main results table source from previously reported results, to cement our work, we have also rerun PET/V-PET under the same splits, seeds, preprocessing and training budget (reported in our response to Reviewer TchU).

About attribution of the gains

Combining these reruns with Table 12, where only λ1\lambda_1 and λ2\lambda_2 vary, separates the gain on DTD (3-shot) into its parts:

DTD (3)Rerun baselineGAFT with λ1=λ2=0\lambda_1=\lambda_2=0Full GAFT
LoRA, PET66.1767.5970.41
LoRA, V-PET67.2568.0369.17
AdaptFormer, PET67.1467.4769.34
AdaptFormer, V-PET68.2168.2969.13

In all four configurations, the two regularizers account for the majority of the total gain, while GAFT with both regularizers disabled is +0.08 to +1.42 above the baseline.

About variability for existing runs and results

We thank the reviewer for the suggestion. To elaborate on the results, we report the mean ± standard deviation for additional datasets, over the three seeds, as follows.

PEFTMethodDTD (3)KITTI (5)RESISC45 (1)CLEVR-C (10)
LoRAPET (rerun)66.17 ± 0.3843.95 ± 0.5955.28 ± 0.1733.08 ± 0.12
PET + GAFT70.41 ± 0.2455.73 ± 0.4162.86 ± 0.2332.05 ± 0.20
V-PET (rerun)67.25 ± 0.3058.6 ± 0.5365.99 ± 0.4536.80 ± 0.75
V-PET + GAFT69.17 ± 0.2761.67 ± 0.3378.95 ± 0.7451.84 ± 0.98
AdaptFormerPET (rerun)67.14 ± 0.3845.85 ± 0.6456.24 ± 0.5533.36 ± 0.54
PET + GAFT69.34 ± 0.2658.68 ± 0.3062.85 ± 0.3237.40 ± 0.41
V-PET (rerun)68.21 ± 0.1959.02 ± 0.2966.24 ± 0.2736.89 ± 0.72
V-PET + GAFT69.13 ± 0.2758.90 ± 0.5175.61 ± 0.4648.14 ± 0.59

On KITTI, where PET + GAFT yields large gains in Table 1, the improvements over our reruns (+11.78 with LoRA, +12.83 with AdaptFormer) exceed the seed standard deviations by more than an order of magnitude. A similar pattern can be observed for runs that show substantial improvements, such as for RESISC45 and CLEVR-C. Overall, almost every other gain in the table also exceeds the sum of the two standard deviations, showing us the empirical benefits of GAFT in fine-tuning for such tasks.

Once again, we thank the reviewer for taking the time to assess our work and acknowledging GAFT's strengths. We hope these comments have fully addressed your concerns.

Reviewer_HU3rOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GAFT proposes a geometry-aware regularisation approach for adapting vision foundation models in low-label regimes using semi-supervised learning (SSL). The method constructs clusters around labeled samples, which act as anchors, and assigns unlabeled samples to these clusters based on cosine similarity in the feature space. It introduces two regularisation terms, alignment and robustness, to encourage consistency between labeled anchors and unlabeled samples and among samples within the same cluster, respectively. The clusters are periodically updated during fine-tuning. The authors also provide a theoretical generalisation bound to motivate the proposed regularisation terms.

The method is evaluated primarily using DINOv2 with LoRA and AdaptFormer on six VTAB datasets, with additional backbone experiments in the main paper and appendix. GAFT is applied on top of existing SSL methods and demonstrates improvements in classification accuracy across most settings. The authors also evaluate robustness under image corruptions and class imbalance. The proposed approach is simple and can be incorporated into existing SSL methods without architectural modifications.

Critical strengths
  1. The proposed method is simple and introduces geometry-aware regularisation terms that can be applied on top of existing SSL methods without architectural modifications. This makes the approach practical and easy to integrate into existing foundation model adaptation pipelines.

  2. GAFT demonstrates improvements over existing SSL methods across multiple datasets and fine-tuning strategies using DINOv2. The results show that the proposed regularisation terms can improve adaptation performance in low-label regimes. The additional experiments on robustness and other foundation models further support the effectiveness of the approach.

Critical weaknesses
  1. Previous SSL and VLM adaptation methods such as PAWS [1] and ClusterAdapter [2] already exploit geometric information for representation learning and foundation model adaptation. PAWS uses similarities between labeled support samples and unlabeled samples to learn representations, while ClusterAdapter uses multiple prototypes for few-shot VLM adaptation. GAFT's alignment and robustness regularizers appear to be natural extensions of existing geometry-based approaches, raising concerns about the methodological novelty.

  2. Table 1 reports results using fixed shot counts for each dataset. Evaluating different values of K for each dataset would provide a better understanding of GAFT's effectiveness under varying levels of label scarcity. The experiments in Table 2 consider only a few datasets. Including additional datasets would help establish the generalizability of the observed improvements.

  3. The individual contributions of the alignment regularizer, robustness regularizer, and anchor risk should be demonstrated through ablation experiments. Additionally, sensitivity analysis of the regularization coefficients λ1\lambda_1 and λ2\lambda_2 would help understand their influence on performance.

  4. The writing could be improved, as the methodological novelty and some details of the proposed approach are not sufficiently clear.

[1] Semi-supervised learning of visual features by non-parametrically predicting view assignments with support samples, ICCV 2021

[2] Tuning Vision-Language Models With Multiple Prototypes Clustering, TPAMI 2024

Questions for authors

Please see weaknesses

4 comments
AuthorsOpenReview ↗

Addressing W1 & W4

We would like to thank the reviewer for acknowledging GAFT's plug-in simplicity and consistent improvements across datasets and PEFT methods. We would like to address your remaining concerns as follows.


W1: "Previous SSL and VLM adaptation methods such as PAWS [1] and ClusterAdapter [2] already exploit geometric information… GAFT's alignment and robustness regularizers appear to be natural extensions of existing geometry-based approaches, raising concerns about the methodological novelty."

W4: "The writing could be improved, as the methodological novelty and some details of the proposed approach are not sufficiently clear."

About PAWS (Assran et al., ICCV 2021)

PAWS is a semi-supervised representation pre-training method: given an unlabeled image, it assigns a soft pseudo-label by computing a similarity-weighted average over all labeled support samples in a mini-batch, then minimizes the cross-entropy between the resulting pseudo-labels of two augmented views of the same image. GAFT differs from PAWS on three following points.

  • Firstly, PAWS replaces the training objective during representation pre-training; GAFT is added on top of an existing task-adapted SSL pipeline (Eq. 10) as an auxiliary plug-and-play term, without modifying the pseudo-labeling step or the base objective.
  • Secondly, in terms of graph structure, PAWS generates a soft distribution over all support samples at the same time (a full bipartite affinity graph). In contrast, GAFT assigns each unlabeled point to exactly one labeled anchor to form a Voronoi-style partition, which is what enables the partition-based theory in Theorems 3.5, 3.7, and A.5. Moreover, GAFT's regularizers do not rely on consistency between augmented views.
  • Thirdly, PAWS sets out to train an encoder from scratch on the unlabeled pool. This is different from GAFT fine-tuning a pretrained VFM through PEFT to leverage DINOv2's or CLIP's innate embedding structure directly (Corollary 3.6).

About ClusterAdapter (Guo et al., TPAMI 2024)

ClusterAdapter is a few-shot VLM adaptation method for CLIP, with no unlabeled pool. It allocates multiple learnable prototypes per class, initializes them from CLIP text embeddings and K-means over the labeled support set, and regularizes them toward frozen CLIP anchors to prevent forgetting. This work is fundamentally different from GAFT.

  • Firstly, ClusterAdapter is purely few-shot supervised, while GAFT is semi-supervised and explicitly exploits an unlabeled pool via LGAFT\mathcal{L}_{\mathrm{GAFT}}.
  • Second, in ClusterAdapter, "anchors" are fixed reference points derived from CLIP's frozen representations, used to prevent prototype drift. In GAFT, anchors are the labeled training samples themselves, which define Voronoi regions and carry the bound-derived weights. While both approaches use the term "anchor", these are not the same.
  • Thirdly, ClusterAdapter keeps the CLIP backbone entirely frozen. In contrast, GAFT uses LoRA or AdaptFormer to update the backbone, so the partition geometry co-evolves with the backbone, which Theorem 3.7 covers.

=> Overall, we would argue that even when other geometry-based techniques have existed, the Voronoi graph construction, the plug-and-play nature, and the explicit semi-supervised VFM fine-tuning task (especially in limited labeled data scenarios) set GAFT apart from previous approaches.

About writing clarity (W4)

Thank you for the suggestion. We have updated the PDF to add Section 3.4 (Theorem 3.7, proof in Appendix A.4) which extends the theory to cover the data-dependent partitions used throughout training, and Appendix A.7 (Theorem A.5) which provides a uniform bound over all predictors h∈Hh\in\mathcal H and all admissible partitions, to directly justify Rrobust\mathcal R_{\rm robust} as a training surrogate. These additions will help to strengthen and elaborate on the methodological novelty and underlying theoretical details of GAFT.

Once again, we would like to thank the reviewer for the constructive feedback. We will address subsequent points in the next Official Comment.


Assran, M., Caron, M., Misra, I., Bojanowski, P., Joulin, A., Ballas, N., Rabbat, M. (2021). Semi-supervised learning of visual features by non-parametrically predicting view assignments with support samples. ICCV.

Guo, M.-H., Zhang, Y., Mu, T.-J., Huang, S. X., Hu, S.-M. (2024). Tuning vision-language models with multiple prototypes clustering. IEEE Trans. Pattern Anal. Mach. Intell., 46(12):11186–11199.

Zhang, P., Mai, Z., Nguyen, Q.-H., Chao, W.-L. (2025). Revisiting semi-supervised learning in the era of foundation models. NeurIPS.

AuthorsOpenReview ↗

Addressing W2 & W3

We would like to continue addressing the reviewer's concerns in this comment.


W2: "Table 1 reports results using fixed shot counts for each dataset. Evaluating different values of K for each dataset would provide a better understanding… The experiments in Table 2 consider only a few datasets. Including additional datasets would help establish the generalizability…"

W3: "The individual contributions of the alignment regularizer, robustness regularizer, and anchor risk should be demonstrated through ablation experiments. Additionally, sensitivity analysis of the regularization coefficients (λ1) and (λ2)…"

About shot counts (W2)

Appendix C.4 (Tables 10 and 11) already reports a shot-count sweep on DTD, varying the labeled budget across 1-, 3-, and 8-shot settings for all four PET/V-PET × LoRA/AdaptFormer configurations.

PEFTMethod1-shot3-shot8-shot
AdaptFormerPET44.68 → 47.34 (+2.66)67.20 → 69.34 (+2.14)72.77 → 74.84 (+2.07)
V-PET49.79 → 57.07 (+7.28)68.20 → 69.13 (+0.93)74.63 → 77.93 (+3.30)
LoRAPET43.62 → 46.54 (+2.92)66.70 → 70.41 (+3.71)72.39 → 75.90 (+3.51)
V-PET49.47 → 56.44 (+6.97)67.80 → 69.17 (+1.37)74.57 → 78.19 (+3.62)

All 12 entries are positive, with the largest gains arising in the 1-shot regime for three of the four configurations. As the shot count increases, the advantage remains consistent (+2.07 to +3.62 at 8 shots). For generalizability across datasets and backbones, we note that Table 1 (main table) already covers six datasets, three for Table 2 (on class imbalance analysis), and cross-backbone results are reported in Table 9 (CLIP ViT-B/16, DTD / RESISC45 / KITTI) and Table 3 (Qwen3-VL).

About ablations and λ sensitivity (W3)

The individual contributions of the two regularizers and the anchor-reweighted supervised term are already reported in Appendix C.6 (Table 12) for DTD (3-shot, DINOv2) across all four configurations. That table isolates Rrobust\mathcal R_{\rm robust} alone, Ralign\mathcal R_{\rm align} alone, and the full GAFT objective against a baseline that retains the anchor-reweighted term while disabling both regularizers (λ1=λ2=0\lambda_1=\lambda_2=0). The regularizers add +0.84 to +2.82 points beyond the anchor-reweighted baseline in every configuration. Combining Table 12 with the rerun baselines reported in our response to Reviewer TchU further isolates the anchor-reweighted supervised term itself, whose contribution on DTD ranges from +0.08 to +1.42 points, with the full decomposition shown below.

DTD (3-shot)Rerun baseline+anchor risk (λ=0\lambda=0)+regularizers (full GAFT)
LoRA, PET66.1767.5970.41
LoRA, V-PET67.2568.0369.17
AdaptFormer, PET67.1467.4769.34
AdaptFormer, V-PET68.2168.2969.13

Sensitivity analysis of (λ1,λ2)(λ_1, λ_2) is reported in Appendix C.5 (Figure 6), which varies five settings on DTD (3-shot, DINOv2) across all four configurations and tracks accuracy over training epochs. Moderate values such as (0.01, 0.1)(0.01,\, 0.1) consistently yield strong accuracy; very small values give suboptimal results, and overly large λ2\lambda_2 degrades performance. AdaptFormer is less sensitive to the choice than LoRA, and both coefficients were fixed at λ1=0.01\lambda_1=0.01, λ2=0.1\lambda_2=0.1 across all datasets without per-dataset tuning (Appendix B.2.5).

Once again, we would like to thank the reviewer for the constructive review. We hope these comments have fully addressed your concerns.

Reviewer_HU3rOpenReview ↗

Response

Thanks for the response by the authors.

The main concern I have still remains that the method is not distinct or well motivated enough compared to PAWS. Hence, I would like to keep my score.

We would like to address the remaining PAWS concern more directly. The two methods solve categorically different problems and cannot substitute for each other.

PAWS performs large-scale semi-supervised representation pre-training from scratch: it requires enough labeled examples per class to form meaningful soft nearest-neighbor pseudo-labels (e.g., 1,281 per class at 10% ImageNet labels). GAFT is a few-shot PEFT fine-tuning method where the total labeled set may be only 47 to 200 examples across all classes. In this regime, PAWS's soft distribution over the entire support set becomes degenerate: with 3 examples per class over 47 classes, the similarity-weighted average collapses toward uniform and provides no signal.

Conversely, GAFT cannot replace PAWS: it is an plug-and-play regularization term added to an existing SSL objective and requires a pre-trained backbone to supply the embedding geometry it conditions on.

The methods therefore are non-overlapping. The fact that both measure similarities between labeled and unlabeled examples no more makes them extensions of each other than softmax classification and kNN are extensions of each other for sharing cosine similarity.

Once again, we thank the reviewer for their time and hope this clarifies the distinction.

Reviewer_TchUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GAFT adds geometry-based regularization to semi-supervised adaptation of vision foundation models. Unlabeled examples are assigned to their nearest labeled anchors in the backbone feature space. Training combines an anchor-weighted supervised loss with anchor-to-sample alignment and within-region prediction consistency, and the partition is refreshed during fine-tuning. A risk bound motivates the objective. The experiments cover six VTAB tasks, multiple backbones, and two PEFT methods.

Critical strengths

The paper reports useful improvements when GAFT is added to PET and V-PET with DINOv2 and CLIP. The appendix includes component ablations, graph-based baselines, and partition-purity measurements. In particular, Table 12 separates the effect of the reweighted supervised term from the two regularizers. These experiments provide evidence that the objective can be useful in practice.

Critical weaknesses
  1. The methodological novelty is limited. In Eqs. 6–7, the alignment term sums prediction differences over anchor–sample pairs, while the robustness term sums them over pairs within the same region. This is closely related to graph-based smoothness and neighborhood consistency, as studied in manifold regularization and teacher-graph methods. Using labeled anchors does not remove the pairwise nature of these constraints. The paper should make clearer what is new in the partition, weighting, or objective, and why that difference matters beyond using a stronger pretrained feature space.

  2. The theoretical advantage is not established clearly. Theorem 3.5 is valid for an arbitrary anchor partition, but its usefulness still depends on the unknown label-dependent residual g_u being small. Remark A.4 explicitly acknowledges that this requires labels to vary slowly within each region. Thus, removing a global manifold assumption does not establish that an arbitrary VFM-induced partition will help. Corollary 3.6 controls prediction variation through feature distances, but does not control variation of the true labels. I would like a clearer account of what guarantee improves over existing geometry-based SSL analyses and when the proposed partition gives an informative bound.

  3. Feature proximity need not imply task-relevant semantic similarity. The purity analysis in Tables 14–15 is useful, but also shows the limitation: on Retinopathy with CLIP, anchor-label agreement rises from 15.74% initially to only 34.09% at epoch 25. Most assignments therefore still connect different classes. The controlled accuracy gain with GAFT is encouraging, but does not explain how the alignment term avoids reinforcing these incorrect relationships. Please analyze the behavior on incorrect anchor assignments and identify when the method helps or hurts as neighborhood quality changes.

  4. The comparisons need stronger controls. The main baselines do use DINOv2, and PET/V-PET are recent VFM-oriented methods, so the concern is not simply a backbone mismatch. However, most main-table baseline values are imported from Zhang et al. (2025), rather than rerun alongside GAFT. The stated matching recipe is helpful, but rerunning the key baselines with identical splits, seeds, preprocessing, and training budgets would make the gains more convincing. Table 17 also provides useful LP/Laplacian comparisons on two tasks. A more direct novelty test would compare GAFT with standard graph regularizers added to the same PET/V-PET objective, while controlling for the extra supervised term and sampling strategy. Table 12 addresses part of this issue on DTD, but a broader matched comparison is still needed.

Questions for authors
  1. What is the main technical difference from an anchor-based graph consistency objective? Can you isolate its benefit by replacing GAFT with a standard graph regularizer inside the same PET/V-PET pipeline?

  2. Under what conditions does the VFM-induced partition make g_u small? How does the method handle incorrect anchor assignments, particularly in the low-purity Retinopathy setting?

  3. Can you rerun the key PET/V-PET and graph-based controls under the same splits, seeds, sampling strategy, and supervised-loss weighting? This would help assess both the size and the source of the improvement.

5 comments
AuthorsOpenReview ↗

Addressing W1 & Q1

We would like to thank the reviewer for acknowledging the various benefits of GAFT. We would like to address your remaining concerns as follows.


W1: "The methodological novelty is limited... This is closely related to graph-based smoothness and neighborhood consistency..."

Q1: "What is the main technical difference from an anchor-based graph consistency objective? Can you isolate its benefit by replacing GAFT with a standard graph regularizer inside the same PET/V-PET pipeline?"

In essence, by utilizing the relationships among data points, GAFT itself can be classified as a graph-based regularizer approach assisting semi-supervised learning. The difference, compared to Manifold regularization (Belkin et al., JMLR 2006) and Teacher-graph consistency, SNTG (Luo et al., CVPR 2018), is in the construction of the graph and the penalty. This leads to major differences in theoretical and practical aspects:

Theory:

  • The kNN or ε-neighbourhood graph with heat-kernel weights graphical model in Belkin's work links an unlabeled point to potentially numerous other samples so long as their distance is under a threshold. In contrast, for GAFT, each unlabeled point is tied to exactly one anchor and interacts only with its own region until the next refresh. This sparser and simpler graph lends itself more readily to the theoretical backing of GAFT in Section 3.2.
  • Moreover, since GAFT refreshes the graph throughout the fine-tuning process, this allows the connection between unlabeled and labeled samples, to be updated dynamically and accurately, unlike the completely static graph in Belkin's Manifold regularization technique. In our PDF, we have also added Section 3.4, and with it, Theorem 3.7 (whose proof is in Appendix A.4), as the theoretical proof as to how a dynamic partition can yield a tighter generalization bound.
  • Furthermore, in Belkin's work, distances are calculated from the data samples themselves, and in Luo's work, the graph is built based on the pseudo-labels from a teacher model. For GAFT, the graph is based on the backbone geometry, so it is able to make use of VFMs' innate embedding capacity to group related data points. Thus, when GAFT is added to a pseudo-labeling method such as PET/V-PET, it contributes a new signal that is not a function of those same pseudo-labels.

Practice:

  • Neither manifold regularization nor SNTG introduce a partitioning-based re-weighted supervised term to their objective. Table 1 and Table 12 together has already demonstrated that adding this re-weighted term already introduces some benefit in terms of the test accuracy.
  • We would like to point out that in Table 17, where we let Label propagation (Iscen et al., CVPR 2019) and Laplacian regularization (Ando & Zhang, NeurIPS 2006) use a kNN graph over the same DINOv2 embeddings. Adding GAFT still improves them by +3.62 / +7.86 points (LP) and +4.04 / +2.08 points (LapReg) on DTD / KITTI.
MethodDTD (3)KITTI (5)
Label Propagation (LP)57.1235.72
LP + GAFT60.7443.58
Laplacian Regularization (LapReg)56.4942.15
LapReg + GAFT60.5344.23

To isolate the benefits of GAFT as the reviewer asks, we have run additional experiments. Specifically, we kept the PET/V-PET pipeline fixed (DINOv2, LoRA, 3-shot DTD + 5-shot KITTI) and added the objectives of Manifold regularization (using kNN Laplacian, LapRLS objective over one-hot labels) and SNTG (using PET/V-PET pseudo-labels for teacher graph) to compare with using GAFT. Results are as follows.

ExperimentDTD (3) & PETDTD (3) & V-PETKITTI (5) & PETKITTI (5) & V-PET
None66.767.842.658.2
With Manifold regularization65.2767.6144.7357.95
With SNTG67.4568.2447.6859.49
With GAFT70.4169.1755.7361.67

From the results, it is clear that empirically, GAFT offers a greater improvement for test accuracy, compared to other graph-based approaches. These results will be added to an additional section in the Appendix to enhance our paper.

Once again, we would like to thank the reviewer for raising these points. We will address subsequent points in the next Official Comment.


Ando & Zhang (2006). Learning on graph with Laplacian regularization. NeurIPS.

Belkin et al., (2006). Manifold regularization: A geometric framework for learning from labeled and unlabeled examples. JMLR.

Iscen et al., (2019). Label propagation for deep semi-supervised learning. CVPR.

Luo et al., (2018). Smooth neighbors on teacher graphs for semi-supervised learning. CVPR.

AuthorsOpenReview ↗

Addressing W2, W3, & Q2

We would like to continue addressing the reviewer's concerns in this comment.


W2: The theoretical advantage is not established clearly. Theorem 3.5 is valid for an arbitrary anchor partition, but its usefulness still depends on the unknown label-dependent residual g_u being small...

W3: Feature proximity need not imply task-relevant semantic similarity... The controlled accuracy gain with GAFT is encouraging, but does not explain how the alignment term avoids reinforcing these incorrect relationships...

Q2: Under what conditions does the VFM-induced partition make g_u small? How does the method handle incorrect anchor assignments, particularly in the low-purity Retinopathy setting?

About gug_u and the choice of dynamic anchor partitioning

Firstly, to answer these points, we have added Theorem 3.7 (Section 3.4, proof in Appendix A.4) regarding data-dependent partitions, which will hold for the partitions that GAFT refreshes in intervals from the adapted backbone. Specifically, the bound in Theorem 3.7 is proved to mathematically hold for partitions that depend on DD or DuD^u, at the explicit cost of TV(Q∗,Q)\mathrm{TV}(Q^*,Q), which essentially represents the deviation against a fixed reference partition Γ∗\Gamma^*.

In other words, an initially incorrect assignment is not fixed for training. The point of dynamically updated partitions is that an assignment is re-decided at each refresh in the adapted geometry, and a refresh that makes regions more label-homogeneous lowers gug_u, while still keeping the bound valid.

To elaborate, as for specifically when gug_u is small, by its definition in Theorem 3.5 it has two parts. The concentration term C(mln⁡4−2ln⁡δ)/uC\sqrt{(m\ln 4-2\ln\delta)/u} vanishes at rate O(u−1/2)O(u^{-1/2}), so it is small whenever the unlabeled pool is large relative to the number of anchors. The label terms L∑iπ^i[d(si,Diu∣y∗)+d(Diu,Xi∣y∗)]L\sum_i\hat{\pi}_i[d(s_i,D^u_i\mid y^{\ast})+d(D^u_i,X_i\mid y^{\ast})] measure how far the labels in each region are from the anchor's label and from each other, and are zero if and only if every region is label-pure. Hence, gug_u is small when u≫mu\gg m and the VFM-induced partition is label-homogeneous. Again, by nature, gug_u does not depend on hh, but directly on the partition, and via Theorem 3.7, we show that by explicitly designing the partition around dynamic updates, the generalization bound is kept valid.

About incorrect anchor assignments (and challenging datasets in general)

Concerning the Retinopathy setting, which is an extremely difficult classification task, it is true that by the end of the training process, a lot of class assignments are not yet correct. However, empirically speaking, if the alignment term reinforced incorrect relationships, wrongly assigned points would stay with their anchors, and purity with GAFT would lag behind purity without it. Table 15 (CLIP, AdaptFormer, V-PET, Retinopathy) shows the opposite at every refresh.

Without / with GAFTEpoch 5Epoch 15Epoch 25
Purity18.06 / 19.3225.70 / 28.0130.29 / 34.09
Test accuracy24.12 / 27.8432.83 / 34.1938.62 / 40.93

Once again, we would like to thank the reviewer for raising these points. We will address subsequent points in the next Official Comment.

AuthorsOpenReview ↗

Addressing W4 & Q3

We would like to continue addressing the reviewer's concerns in this comment.


W4: "The comparisons need stronger controls... most main-table baseline values are imported from Zhang et al. (2025), rather than rerun alongside GAFT... rerunning the key baselines with identical splits, seeds, preprocessing, and training budgets would make the gains more convincing..."

Q3: "Can you rerun the key PET/V-PET and graph-based controls under the same splits, seeds, sampling strategy, and supervised-loss weighting?"

Thank you for the suggestion. Regarding the graph-based controls, we have added Manifold regularization (Belkin et al., 2006) and SNTG (Luo et al., 2018) to the same PET/V-PET pipeline, as reported in our previous comment under Q1, alongside the controlled Label Propagation and Laplacian regularization comparisons of Table 17.

We would also note that the anchor-reweighted supervised term and the labeled-first, cluster-conditioned sampling strategy are components of GAFT only. A standard graph regularizer (e.g. Manifold regularization, SNTG, etc.) do not contain such components. What can be measured is how much each component contributes, and the paper already isolates both. In Table 12, both regularizers are disabled (λ1=λ2=0\lambda_1=\lambda_2=0) while the reweighted term and sampling are kept, and the regularizers then add +0.84 to +2.82 points on DTD. In Table 8, the two sampling strategies are compared with everything else fixed.

Regarding the imported baselines, we have rerun PET, V-PET and FixMatch with the same labeled splits, seeds, preprocessing and training budget as GAFT (Appendix B.2). Below, the values reported by Zhang et al. (2025) are shown alongside our reruns, together with the standard deviations of the GAFT variants, all over three seeds.

PEFTMethodDTD (3)KITTI (5)
LoRAFixMatch (reported)57.414.8
FixMatch (rerun)59.95 ± 0.5814.86 ± 0.57
PET (reported)66.742.6
PET (rerun)66.17 ± 0.3843.95 ± 0.59
PET + GAFT70.41 ± 0.2455.73 ± 0.41
V-PET (reported)67.858.2
V-PET (rerun)67.25 ± 0.3058.6 ± 0.53
V-PET + GAFT69.17 ± 0.2761.67 ± 0.33
AdaptFormerFixMatch (reported)59.519.8
FixMatch (rerun)60.11 ± 0.3519.04 ± 0.21
PET (reported)67.245.2
PET (rerun)67.14 ± 0.3845.85 ± 0.64
PET + GAFT69.34 ± 0.2658.68 ± 0.30
V-PET (reported)68.259.2
V-PET (rerun)68.21 ± 0.1959.02 ± 0.29
V-PET + GAFT69.13 ± 0.2758.90 ± 0.51

Our reruns closely match the reported values, so the comparisons in Table 1 are dependable. As observed, the gains from incorporating GAFT are stable across different seeds, for different datasets and PEFT methods. These results will be added to the Appendix to enhance our paper.

Once again, we thank the reviewer for taking the time to assess our work and acknowledging GAFT's strength. We hope these comments have fully addressed your concerns.


Ando, R. K., Zhang, T. (2006). Learning on graph with Laplacian regularization. NeurIPS.

Belkin, M., Niyogi, P., Sindhwani, V. (2006). Manifold regularization: A geometric framework for learning from labeled and unlabeled examples. JMLR.

Iscen, A., Tolias, G., Avrithis, Y., Chum, O. (2019). Label propagation for deep semi-supervised learning. CVPR.

Luo, Y., Zhu, J., Li, M., Ren, Y., Zhang, B. (2018). Smooth neighbors on teacher graphs for semi-supervised learning. CVPR.

Zhang, P., Mai, Z., Nguyen, Q.-H., Chao, W.-L. (2025). Revisiting semi-supervised learning in the era of foundation models. NeurIPS.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.