acceptodds
Under review as a conference paper at ICLR 2027

Spectral Gradient Merging for Stable Few-Shot Adaptation of Vision-Language Models

Abstract

Few-shot adaptation of vision–language models remains sensitive to the learning rate and training duration, even when SGD achieves strong accuracy. We propose OLM-SGD, an optimizer that combines class-unique minibatching with orthogonal low-rank class-specific gradient merging. Each minibatch contains at most one example per class. For each parameter tensor, OLM-SGD retains dominant per-example singular components and merges them through polar reconstruction. It directly adapts the original encoder blocks without additional trainable modules or optimizer moments. Our analysis establishes retained-gradient alignment and energy control, with conditional stationarity guarantees and finite-shot population bounds that account for support-set reuse. Experiments span 11 datasets, three CLIP backbones, and five shot levels. OLM-SGD improves dataset-average accuracy over the strongest evaluated parameter-efficient or frozen-feature baseline in all 15 backbone–shot settings, with gains up to 1.9 percentage points. One-shot experiments over 300 epochs and four learning rates also show reduced learning-rate sensitivity and improved accuracy retention. At learning rate , the mean peak-to-final accuracy drop on ViT-B/16 decreases from 6.49 percentage points with SGD to 2.22 with OLM-SGD. Ablations show that polar reconstruction contributes beyond class-unique sampling and low-rank truncation alone. These results support class-specific gradient merging as an effective approach to accurate and stable few-shot adaptation. Our code is included in the supplementary materials and will be released upon publication.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
few-shot learning, vision-language models, CLIP adaptation, full fine-tuning, optimization, gradient merging, low-rank gradients, training stability, learning-rate robustness

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_B3CEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OLM-SGD for few-shot adaptation of vision-language models. It combines class-unique minibatching with tensor-wise low-rank decomposition and polar reconstruction of per-example gradients, directly fine-tuning the Transformer blocks of both CLIP encoders without additional modules or Adam-style optimizer states. The analysis provides retained-gradient energy and alignment bounds, conditional stationarity guarantees, and finite-sample population bounds.

Critical strengths
  1. The paper addresses learning-rate and training-duration sensitivity in few-shot full fine-tuning of vision-language models, a problem with clear practical relevance.
  2. The method is specified in reasonable detail and evaluated across 11 datasets, three CLIP backbones, and five shot settings.
Critical weaknesses
  1. The main comparisons use 70 epochs for OLM-SGD versus 30 for most baselines. For K > 1, OLM-SGD also performs K * ceil(C / B) updates per epoch, compared with ceil(C / B) for its class-mean variant, without compensating for this difference. These mismatches confound the contribution of per-example polar merging. Comparisons under matched update counts, backward-pass counts, or wall-clock budgets, including a class-mean control with the same update count, are needed.
  2. OLM-SGD requires per-example backward passes, tensor-wise SVDs, and two polar decompositions per tensor. Maintaining no optimizer moments does not establish low training cost. The paper lacks FLOPs, peak GPU memory measurements, and overhead analyses across batch sizes, class counts, and backbone scales, limiting assessment of its practical usability.
Questions for authors
  1. Lemma 4.1 guarantees alignment with the retained-gradient mean, not the full empirical or population gradient. Subsequent guarantees require additional assumptions, and Appendix E.19 demonstrates an ascent step even for a smooth, strongly convex objective. Reporting epsilon_tail, epsilon_polar, update-to-full-gradient cosine similarity, and whether the assumptions hold during actual training would clarify the empirical relevance of the theory.
2 comments
AuthorsOpenReview ↗

Response to Reviewer B3C

We thank the reviewer for this constructive comment.

C1. Matched training budgets and the class-mean control

Matched updates with SGD. Appendix H.2 compares OLM-SGD and class-unique SGD on the same trainable parameters at η=10−4\eta=10^{-4}. Both methods match scheduled updates and example counts. Across 11 one-shot datasets on ViT-B/16, OLM-SGD improves mean test accuracy by 0.13, 0.83, and 1.31 percentage points at epochs 30, 70, and 300, respectively. At 16 shots and 70 epochs, OLM-SGD reaches 76.50% across EuroSAT, FGVC-Aircraft, and Stanford Cars, compared with 74.85% for class-unique SGD and 74.74% for random-minibatch SGD. These results average three seeds.

Matched updates with class-mean merging. Appendix H.5 compares the variants on four datasets with ViT-B/16, r=3r=3, and three seeds. At equal support-set exposure (70 passes), per-example merging improves mean accuracy by 1.25 points at 4 shots and 2.55 points at 8 shots. At equal update counts, per-example and class-mean merging reach 66.12% and 66.19% at 4 shots, and 70.94% and 71.42% at 8 shots. Per-example merging uses 70 passes. Class-mean merging uses 280 and 560, respectively.

Matched backward passes. Appendix H.10 isolates the polar step. Both variants compute and truncate the same number of per-example gradients per update. Replacing the arithmetic mean of the truncated gradients with joint polar reconstruction improves accuracy in all 12 dataset–shot settings. At 16 shots, mean accuracy increases from 74.31% to 75.54%.

C2. Training time, memory, and computational scaling

Appendix J profiles SUN397 on one NVIDIA H100 with ViT-B/16 and batch capacity 64. At one shot, OLM-SGD, SGD, and rank-2 LoRA take 89.06, 2.28, and 2.47 seconds per support pass. Peak allocations are 15.30, 14.39, and 15.16 GiB, respectively. OLM-SGD takes 176.24 seconds per pass at two shots and 348.04 at four. Its peak allocation stays at 15.30 GiB. This measures scaling with support size at fixed class count, batch capacity, and backbone.

Cost per update. In one 64-example update, truncated SVDs take 8.31 seconds (54% of update time), per-example backward passes take 4.89 seconds, and factor assembly and polar decomposition take 1.46 seconds. Appendix C.1 counts CKCK gradient decompositions and 2K⌈C/B⌉2K\lceil C/B\rceil polar decompositions per connected trainable tensor per pass. For a tensor of shape mW×nWm_W\times n_W, the concatenated factors have mWm_W and nWn_W rows and Rt=rWBtR_t=r_WB_t columns. When Rt≤min⁡(mW,nW)R_t\leq\min(m_W,n_W), the two reduced SVDs in the polar step cost O((mW+nW)Rt2)O((m_W+n_W)R_t^2) operations per tensor update. The end-to-end profile reports time and peak memory. End-to-end FLOPs and cross-backbone timings remain unmeasured.

Accuracy at equal training time. SGD leads at all three shot counts at 667 seconds. At 2,668 seconds, OLM-SGD leads SGD by 1.00, 0.56, and 0.28 percentage points at one, two, and four shots, respectively. Its gains over LoRA are 5.16, 4.10, and 3.47 points. Both budgets were fixed before evaluation (SUN397, seed 1).

Q1. Alignment and reconstruction along training trajectories

Appendix H.11 measures alignment with ∇F(Θt)\nabla F(\Theta_t) computed over the entire one-shot support set. For EuroSAT, FGVC-Aircraft, and Stanford Cars on ViT-B/16, the 36 initial configurations share nine dataset–seed starts across four rates. We also probed 52 trained checkpoints from 19 trajectories. The update g^t\widehat g_t has positive inner product with the full gradient on 447 of 448 minibatches, including 279 of 280 at trained checkpoints. At epoch 300 and η=10−5\eta=10^{-5}, the mean cosines on full-size minibatches are 0.201 on FGVC-Aircraft and 0.179 on Stanford Cars across three seeds.

Reconstruction errors. On Stanford Cars at η=10−4\eta=10^{-4} and Bt=64B_t=64, we normalize each error by ∣∇F(Θt)∣2\\|\nabla F(\Theta_t)\\|_2 before averaging over minibatches and seeds. At initialization and epoch 300, the normalized truncation bound εt,tail\varepsilon_{t,\mathrm{tail}} is 5.058 and 4.668, respectively. The normalized polar reconstruction error εt,polar\varepsilon_{t,\mathrm{polar}} is 1.562 and 1.549. The absolute batch merge error ∣g^t−gt∣2\\|\widehat g_t-g_t\\|_2 falls from 55.93 to 0.819.

Theoretical conditions and observed descent. All 448 minibatches satisfy the retained-alignment and energy inequalities in Lemma 4.1. Both sufficient lower bounds for full-gradient alignment are nonpositive on all draws. The measured inner products describe individual draws. The stationarity assumptions concern conditional expectations over draws at fixed parameters. Finite-step trials begin at identical parameters. At η=10−3\eta=10^{-3} and the same displacement norm, the merged direction reduces the full support loss at all nine initial states. The raw and truncated means increase it at all nine.

Reviewer_B3CEOpenReview ↗

Response to the authors

Thanks for the response and additional results. I would like to keep my current score.

Reviewer_Vf9rOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OLM-SGD for few-shot CLIP adaptation, constructing class-unique minibatches, computing gradients for individual examples, applying tensor-wise low rank SVD truncation and merging the retained factors through polar reconstruction, updating the original Transformer blocks of both encoders. Experiments are conducted across 11 datasets, three backbones, and five shot levels, and include long-horizon trajectories and component ablations. The analysis establishes retained-gradient alignment and energy bounds with conditional optimization and population guarantees.

Critical strengths
  • reducing learning rate and stopping time sensitivity is valuable when few-shot supervision leaves little room to validate

  • the paper includes broad accuracy comparisons, long-horizon experiments, polar-removal ablations and direction–norm interventions

  • the authors acknowledge dataset-specific failures, conditional convergence assumptions and counterexamples around ascent directions and discontinuous merges

  • matched polar-removal experiments indicate that low rank truncation alone does not replicate the full method's accuracy and retention

Critical weaknesses
  1. Novelty relative to iterative merging (ATM). The spectral merge is taken directly from TSV-M, which the paper acknowledges. My larger concern is with the claim that turning a static merging rule into an online optimizer is itself a contribution. ATM (Zhou et al., 2024) already shows that a one-epoch task vector equals a scaled negative gradient. It frames merging as one step of an alternating tune-and-merge procedure and explicitly allows any interference-resolution operator in the merge step. Suppose each support example is treated as its own "task", each task takes a single gradient step under the full CC-way loss, and TSV-M with mean aggregation is used as the merge. As far as I can tell, this recovers the OLM-SGD update exactly, since the polar factors are scale-invariant. Neither ATM nor the follow-up "On Task Vectors and Gradients" (Zhou et al., 2025) is cited. Please discuss both, and state precisely what is new beyond the per-example granularity and the few-shot CLIP setting.

  2. The most relevant baseline is in the appendix. Table 1 and the abstract compare against PEFT and frozen-feature methods. For a full fine-tuning optimizer, however, the natural comparison is plain SGD on the same parameters, which only appears in Appendix H.1. There the gap is much smaller: 0.3–0.9 points on ViT-B/32 and 0.5–1.2 on ViT-B/16. On ViT-L/14, SGD is actually better at 1 shot (77.8 vs. 77.4), and OLM-SGD leads by only 0.1–0.5 at the other shot levels. These fixed-schedule results also give SGD 30 epochs and OLM-SGD 70. The 300-epoch runs help, but they only cover 1-shot ViT-B/16. I would move SGD into the main table and either match the training budgets or tune every method's schedule under the same protocol.

Separately, some baseline numbers look off. One-shot GDA is roughly 17 points below zero-shot CLIP on all three backbones. I suspect the shared covariance becomes degenerate at K=1K=1 under deterministic preprocessing, in which case the ten feature-extraction passes in Table 8 would produce identical features. Could the authors check this?

  1. Computational cost. Each update in Algorithm 2 requires up to 64 separate backward passes, a truncated SVD per example and per tensor, and two further SVDs per tensor for the polar step. Every example's loss also depends on all CC class prototypes. Each backward pass therefore runs through the text encoder for every class, which on ImageNet means 1,000 prompts. Counting decompositions, as in Appendix C.1, does not tell me what this costs in practice.

Please report wall-clock time and peak memory relative to SGD and LoRA, ideally for ViT-L/14 on 16-shot ImageNet, and compare accuracy under matched compute. Since the margin over SGD is often below one point, this largely determines whether the method is worth using. Not storing optimizer moments does not by itself make training cheap.

  1. Learning-rate robustness vs. effective step size. Lemma 4.1 gives ∥g^t∥22≤Bt−2∑b∥gt(b)∥22\|\hat g_t\|_2^2 \le B_t^{-2} \sum_b \|g_t^{(b)}\|_2^2. When per-example gradients are positively correlated, the OLM update can therefore be considerably smaller than the SGD mean. At the same nominal learning rate, the two methods may be taking quite different effective steps, so a flatter accuracy-vs-learning-rate curve could partly be a shifted one. The 10−510^{-5} results fit this reading: OLM-SGD adapts more slowly early on, and SGD at that rate ends at 71.46% versus 71.75% for OLM-SGD.

The direction/norm swap in Figure 1(b) is the strongest evidence that the merged direction matters. However, it covers only three datasets, with three, two and one seeds respectively. I would like to see:

  • norm-matched comparisons on all datasets;
  • update-norm trajectories over training;
  • learning-rate grids calibrated per optimizer and extending below 10−510^{-5}.

Muon and PolarGrad (arXiv:2505.21799) are also natural baselines, since both orthogonalize the update and PolarGrad additionally rescales it by the nuclear norm.

  1. Several component claims are not isolated.
  • Class-unique sampling. The long-horizon sampling comparison uses 1-shot data, where ordinary shuffled minibatches are already class-unique. It therefore says nothing about the benefit of class-unique sampling at higher shot counts.
  • Class-mean ablation. OLM-SGD makes KK times as many updates per epoch at the same learning rate, so the comparison mostly reflects the number of steps.
  • GaLore control. It mixes aggregated-gradient projection with Adam-style moments. The cleaner controls, SGD with a truncated mean gradient and a polar update of the aggregated gradient, are missing.
  • Inconsistent configuration. Appendix D gives 10−410^{-4} for class-unique SGD, while Appendix H gives 10−310^{-3}. H.2 also does not contain the fixed-schedule results it is cited for.
  1. The theory does not explain the empirical gains. Lemma 4.1 establishes alignment with the retained (truncated) mean, not with the full objective gradient. The stationarity results either carry a reconstruction-error term or rely on alignment conditions that are never checked on the actual runs. The authors are upfront about this, which I appreciate. Still, the theory currently does not account for the two things the paper cares most about: better generalization and stability. Measuring the actual alignment, the reconstruction error and the conditioning of the concatenated factors along real trajectories would make it much more informative.

A related point: Roulet & Agarwala (2025) show that where a nonlinear transformation is placed relative to averaging matters, and that applying it per example can hurt the signal-to-noise ratio. This does not mean spectral merging must fail here, but it is a good reason to explain why it works.

Questions for authors
  1. What distinguishes OLM-SGD algorithmically from an ATM-style procedure using singleton subtasks, one local gradient step and TSV-M merging?

  2. What are the training time and peak memory across backbones, and do the gains persist under matched wall-clock budgets?

  3. How does OLM-SGD compare with Muon, PolarGrad and SGD-based spectral controls applied after gradient averaging?

  4. Do the class-mean results persist after controlling for update count or cumulative step size? Can the independent contribution of class-unique sampling be tested at K>1K>1?

  5. What explains the one-shot GDA degradation, and which learning rate and results correspond to the class-unique SGD fixed-schedule experiment?

  6. Can you report actual update scales, full-gradient alignment and reconstruction errors along training trajectories?

6 comments
AuthorsOpenReview ↗

Response to Reviewer Vf9r (1/6)

We thank the reviewer for the careful assessment. We address the concerns below.

C1. Novelty relative to ATM and TSV-M

We thank the reviewer for highlighting ATM (Zhou et al., 2024) and On Task Vectors and Gradients (Zhou et al., 2025), which primarily investigate task-vector merging across multiple tasks. Our work addresses a distinct question: can a single few-shot classification task benefit from multitask-inspired merging of class-specific gradients? To our knowledge, OLM-SGD is the first approach to investigate this perspective for few-shot CLIP adaptation. Unlike multitask model merging, we consider one classification task on a single dataset, where each instance gradient is computed using the full multiclass loss.

We study three questions:

  1. Gradient aggregation: Can preserving distinct class-specific instance gradients through orthogonal low-rank merging improve discriminative adaptation compared with conventional gradient averaging?
  2. Optimization mechanism: Does modifying the gradient direction through polar reconstruction provide benefits beyond low-rank truncation and update-norm control?
  3. Optimization robustness: Can structured gradient merging reduce sensitivity to learning rates and training duration, thereby enabling reliable few-shot adaptation under validation-free protocols?

Algorithmically, OLM-SGD combines class-unique minibatching with tensor-wise spectral merging of individual gradients at each optimization step. It directly adapts both CLIP encoders without additional trainable modules or adaptive optimizer states. We acknowledge the mathematical connection to TSV-M with an ATM-style construction with singleton subtasks and one-step updates. However, neither TSV-M nor ATM investigate class-unique gradient merging within a single multiclass objective or its implications for few-shot adaptation robustness.

Theoretically, we establish retained-gradient alignment and energy bounds, conditional stationarity guarantees, and finite-shot population bounds accounting for repeated support-set use. These results characterize the spectral merging mechanism within a stochastic optimization framework rather than as a post-training model-merging operation.

Empirically, across 11 datasets, three CLIP backbones, and five shot settings, OLM-SGD demonstrates accuracy improvements over the evaluated baselines. Controlled low-rank truncation and polar-reconstruction ablations, together with direction–norm interventions, establish the importance of the proposed gradient-merging mechanism. Beyond accuracy, extensive learning-rate sweeps, 300-epoch training trajectories, and cross-dataset epoch-transfer experiments demonstrate improved learning-rate robustness, long-horizon stability, and reduced sensitivity to training duration. These properties are particularly valuable for validation-free few-shot adaptation, where reliable hyperparameter selection and early stopping are challenging.

C2. Full fine-tuning SGD baseline and matched training budgets

We agree that full-parameter SGD is the primary optimizer baseline. The existing comparison in Appendix H.1 uses the same trainable CLIP parameters, classification objective, and learning rate (10−410^{-4}), across 11 datasets, three backbones, five shot levels, and three seeds. OLM-SGD leads SGD in 14 of the 15 backbone–shot averages. Its gains are 0.3–0.9 points on ViT-B/32 and 0.5–1.2 points on ViT-B/16. On ViT-L/14, it leads by 0.1–0.5 points at 2–16 shots. The revised abstract reports the comparison with SGD. Table 1 of the manuscript includes SGD across all 15 settings.

Why 30 epochs for SGD and 70 for OLM-SGD? The original evaluation applied these method-specific schedules unchanged across datasets, backbones, and shot levels, rather than selecting a separate stopping epoch for every benchmark. Table 2 of the manuscript compares the fixed schedules with the peaks of the 300-epoch one-shot trajectories. At 10−410^{-4}, the dataset-average SGD curve peaks at epoch 21 (71.61%), while its fixed epoch-30 accuracy is 71.49%, only 0.12 points below the peak. OLM-SGD peaks at epoch 67 (72.05%), while its epoch-70 accuracy is 72.02%, only 0.03 points below the peak. Even at their respective retrospective peaks, OLM-SGD retains a 0.44-point advantage.

Table 4. Original schedules and retrospective long-horizon peaks (1-shot ViT-B/16, 11 datasets, seeds 1–3, η=10−4\eta=10^{-4}). Accuracy is in percent; gaps and best-to-final drops are percentage points.

Method    Fixed epoch/acc.   Peak epoch/acc.   Gap   E300 acc.   Drop
SGD          30 / 71.49        21 / 71.61     0.12     70.46    1.81
OLM-SGD       70 / 72.02        67 / 72.05     0.03     71.75    0.95
AuthorsOpenReview ↗

Response to Reviewer Vf9r (2/6)

C2 (continued). Fixed schedules and matched horizons

Peak epochs maximize the dataset- and seed-averaged test-accuracy trajectory. Best-to-final drops are computed within each dataset–seed trajectory before averaging. Peak selection here is a retrospective diagnostic, not a validation-based tuning procedure. The smaller 300-epoch degradation (0.95 versus 1.81 points) further supports OLM-SGD's robustness to training duration.

Matched training horizons. We evaluated class-unique SGD (SGD-CU) at epoch 30 in 495 runs. Nine 16-shot ViT-B/16 runs continued to epoch 70. The matched comparisons use the same learning rate, support-set exposure, and number of scheduled steps for SGD and rank-3 OLM-SGD. Table 5 reports dataset-level one-shot ViT-B/16 accuracy at epochs 30, 70, and 300. The 16-shot comparison uses EuroSAT, FGVC-Aircraft, and Stanford Cars at epoch 70. All results average three seeds.

Table 5. Dataset-level matched-horizon top-1 test accuracy (%) on ViT-B/16 at η=10−4\eta=10^{-4}. All entries are means over three seeds; OLM-SGD uses r=3r=3.

(a) One shot across all 11 datasets

Epoch 30
Dataset          SGD-R  SGD-CU  OLM-SGD
Caltech101       94.27  94.37*    94.05
DTD             53.17*  53.17*    53.03
EuroSAT          68.08   68.08   69.27*
FGVC-Aircraft    29.38   29.37   30.42*
Food101         85.91*   85.91    85.82
ImageNet         67.37   67.46   69.30*
Oxford Flowers   82.46  82.47*    81.40
Oxford Pets     92.60*   92.58    92.30
Stanford Cars   69.24*   68.91    68.67
SUN397           69.68   69.72   70.12*
UCF101           74.20  74.21*    73.35
Mean (11)        71.49   71.48   71.61*

Epoch 70
Dataset          SGD-R  SGD-CU  OLM-SGD
Caltech101      94.23*   94.16    94.02
DTD              53.53   53.55   53.90*
EuroSAT          68.34   68.33   69.73*
FGVC-Aircraft    28.68   28.78   30.82*
Food101          85.92   85.92   85.95*
ImageNet         64.30   64.54   67.71*
Oxford Flowers   83.12   83.16   83.30*
Oxford Pets      92.53   92.53   92.67*
Stanford Cars    68.43   68.31   69.59*
SUN397           69.50   69.47   70.26*
UCF101          74.37*   74.36    74.31
Mean (11)        71.18   71.19   72.02*

Epoch 300
Dataset         SGD-R  SGD-CU  OLM-SGD
Caltech101      94.01   94.00   94.10*
DTD             53.61   53.59   54.26*
EuroSAT         68.29   68.30   70.33*
FGVC-Aircraft   28.88   28.58   29.70*
Food101         85.89   85.90   86.02*
ImageNet        56.88   56.96   62.48*
Oxford Flowers  83.65   83.73   85.02*
Oxford Pets     92.52   92.53   92.80*
Stanford Cars   67.69   67.68   69.47*
SUN397          69.19   69.17   70.10*
UCF101          74.45   74.41   74.98*
Mean (11)       70.46   70.44   71.75*

(b) Sixteen shots on three datasets at 70 epochs

Dataset        SGD-R  SGD-CU  OLM-SGD
EuroSAT        93.69  93.84*    93.24
FGVC-Aircraft  48.87   48.87   52.86*
Stanford Cars  81.66   81.82   83.41*
Mean (3)       74.74   74.85   76.50*

SGD-R: random minibatching; SGD-CU: class-unique minibatching. The 16-shot SGD-R entries use the newly completed three-seed matched-batch experiments. An asterisk marks the highest accuracy within each dataset and horizon. At one shot, ordinary random sampling is already class-unique. OLM-SGD improves on SGD-CU by 0.13, 0.83, and 1.31 points at the three matched horizons, respectively. At epoch 300, OLM-SGD also achieves higher dataset-level mean accuracy on all 11 datasets. At 16 shots and 70 epochs, OLM-SGD improves the three-dataset mean over SGD-R by 1.76 points (76.50% versus 74.74%) and over SGD-CU by 1.66 points (76.50% versus 74.85%), including gains of 3.98 points on FGVC-Aircraft and 1.58 points on Stanford Cars relative to SGD-CU. Appendix H.2 of the manuscript reports both matched-horizon comparisons.

Optimization robustness. At 10−310^{-3}, 300-epoch OLM-SGD reaches 69.23% versus 59.68% for SGD-CU. Its cross-dataset epoch-transfer diagnostic reduces the target-oracle gap from 1.50 points for SGD to 0.91 points. In the separate matched-wall-time SUN397 experiment, OLM-SGD improves over SGD by 1.00, 0.56, and 0.28 points at K∈1,2,4K\in\\{1,2,4\\} under the same 2,668-second training budget (one seed, Appendix J).

Together, the original near-peak schedules and the new matched-horizon and matched-time controls show that OLM-SGD's contribution extends beyond a longer original training schedule. Its principal benefit is improved accuracy retention and reduced sensitivity to learning rate and training duration, which are especially valuable in validation-free few-shot adaptation.

AuthorsOpenReview ↗

Response to Reviewer Vf9r (3/6)

C3. One-shot GDA baseline

The reviewer is correct about the covariance. At one shot, deterministic preprocessing yields one distinct support feature per class. Repeating feature extraction leaves the within-class scatter zero in exact arithmetic. With one extraction per image, the reference GDA estimator reproduced the zero-shot predictions in all 90 deterministic fits across ten datasets, three backbones, and three seeds.

The one-shot GDA entries in manuscript Table 1 use the ten-pass procedure with the numerical fallback specified in Appendix D (Table 9). We reproduced its accuracy deficit on Food101 (ViT-B/16, seed 1). Repeated extraction produced identical features, but their floating-point mean left a small residual. The fallback amplified the resulting class-dependent scores. On the same support features, the ten-pass procedure gave 43.38% accuracy. The one-extraction reference estimator and zero-shot CLIP both gave 85.23%. Across the ten deterministic datasets, the reference estimator matches zero-shot CLIP. On ViT-B/16, the corresponding mean accuracies are approximately 63.8% for this reference and 72.5% for one-shot OLM-SGD, using the per-dataset results in Appendix G.

C4. Computational cost and matched wall-clock comparison

We profiled OLM-SGD (r=3r=3), SGD, and rank-2 LoRA [1] trained with Adam on SUN397 using CLIP ViT-B/16. All runs used the same NVIDIA H100 GPU, seed 1, a fixed learning rate of 10−410^{-4}, and K∈1,2,4K\in\\{1,2,4\\}.

Training cost and GPU memory. Table 1 reports measured steady-pass time and peak allocated training memory for K=1K=1. OLM-SGD fine-tunes the same 122.88 million parameters as SGD while retaining a comparable GPU memory footprint. Its peak allocation is 15.30 GiB, only 0.91 GiB above SGD and 0.14 GiB above LoRA, which trains 0.55 million parameters.

Table 1. Training cost and peak allocated memory (K=1K=1, one H100 GPU).

Method         Trainable (M)  Peak memory (GiB)  Time/pass (s)
SGD                   122.88              14.39           2.28
LoRA (rank 2)           0.55              15.16           2.47
OLM-SGD (r=3)         122.88              15.30          89.06

One-shot OLM-SGD takes about 39 times as long per support pass as SGD. In one instrumented 64-example update, individual-gradient computation took 4.89 s, truncated SVDs took 8.31 s, and factor assembly and polar decomposition took 1.46 s. At one, two, and four shots, OLM-SGD took 89.06, 176.24, and 348.04 s per support pass. Peak allocated memory stayed at 15.30 GiB. These measurements characterize support-size scaling on ViT-B/16 at fixed batch capacity and class count. We have no corresponding ViT-L/14 measurement for 16-shot ImageNet.

Accuracy under matched wall-clock budgets. We fixed budgets of 667, 1,334, and 2,668 s before accuracy evaluation. The upper budget corresponds to approximately 30 one-shot OLM-SGD passes. We selected checkpoints by elapsed training time and timed evaluation and checkpoint writing separately. SGD led at all three shot counts at 667 s. At 2,668 s, OLM-SGD led SGD by 1.00, 0.56, and 0.28 percentage points and LoRA by 5.16, 4.10, and 3.47 points (Table 2).

Table 2. SUN397 test accuracy (%) at a matched training-time budget of 2,668 s.

Shots (K)    SGD  LoRA (rank 2)  OLM-SGD (r=3)  Gain over SGD
1          69.10          64.94         70.10           +1.00
2          70.85          67.31         71.41           +0.56
4          72.11          68.92         72.39           +0.28

Appendix J reports per-pass costs and accuracy at each prespecified time budget.

[1] Low-Rank Few-Shot Adaptation of Vision-Language Models, Zanella and Ben Ayed, CVPRW 2024.

Reviewer_9vwDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to improve optimization for few-shot adaptation of vision-language models. The authors propose OLM-SGD, which first constructs minibatches containing at most one example per class and computes individual gradients separately. Unlike SGD, which directly averages these gradients, OLM-SGD applies SVD to retain the dominant components, followed by polar reconstruction to reduce overlap between the retained singular directions. The resulting merged gradients are then used to update the CLIP encoders. Experimental results show improvements in few-shot accuracy and training stability over the evaluated baselines.

Critical strengths
  • The paper investigates a different approach to optimizing few-shot adaptation of vision-language models by processing individual gradients before merging them. The experiments suggest that modifying gradient direction can be more beneficial than simply controlling gradient magnitude.
  • The method is evaluated across multiple relevant benchmarks, CLIP backbones, and few-shot settings, showing consistent improvements over the evaluated baselines. The ablation studies further support the effectiveness of the proposed method and its individual components.
Critical weaknesses
  1. The motivation for reducing overlap between class-specific gradient directions is not sufficiently justified. It is unclear why overlapping directions should be considered harmful, as different classes may share useful information. The theoretical analysis does not explain why orthogonalizing these directions should lead to better optimization than simply averaging the gradients.

  2. The paper compares individual-gradient merging with class-mean gradient merging. However, individual-gradient merging performs more optimization steps per epoch, making it unclear whether the improvement comes from preserving individual gradients or simply performing more updates. A comparison with the same number of optimization steps would help clarify this.

  3. OLM-SGD requires computing individual gradients and performing SVD and polar decomposition, which may introduce computational overhead. Given the relatively modest accuracy improvements over the strongest baselines in Table 1, it is unclear whether these gains justify the additional cost. A comparison of training time, GPU memory usage, and computational overhead would help assess the practical efficiency of the proposed approach.

Questions for authors
  • The evaluation is limited to few-shot adaptation. Does OLM-SGD also generalize to standard full-data fine-tuning or pretraining?

  • The proposed method updates the transformer blocks of both CLIP encoders. Could the gradient-merging mechanism also be applied to parameter-efficient methods such as LoRA, potentially achieving similar improvements with fewer trainable parameters?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 9vwD (1/3)

We sincerely thank the reviewer for the careful assessment. We address your concerns below.

C1. Motivation for reducing gradient overlap

Appendix H.10 isolates the effect of reducing overlap. Both variants truncate each example gradient to the same rank. Averaging the truncated gradients gives 74.31% mean test accuracy at 16 shots. Joint polar reconstruction raises it to 75.54% and improves all 12 dataset–shot averages across four datasets and three shot counts.

Classes can share useful gradient directions. Each example in OLM-SGD still contributes the gradient of the full multiclass loss. When several examples reinforce the same singular direction, their arithmetic mean gives that direction greater weight. OLM-SGD retains each example's leading singular values and uses the whole minibatch to determine the directions in which those values contribute. Lemma 4.1 shows that the merged update has a positive inner product with the nonzero mean of the truncated gradients under exact decomposition. Thus, the update preserves a component along the direction favored by that mean.

We also measured the change in direction on identical gradients at initialization. Averaging individually truncated gradients gives a mean cosine of 0.905 with SGD. Joint reconstruction gives 0.466. The accuracy comparison above shows that this change improves adaptation under the tested conditions.

C2. Per-instance versus class-mean gradient merging

At equal update counts, per-instance merging reaches comparable accuracy with one quarter as many example presentations at 4 shots and one eighth as many at 8 shots. Appendix H.5 compares ViT-B/16 on EuroSAT, FGVC-Aircraft, Stanford Cars, and DTD across three seeds. Both variants use the same trainable parameters, learning rate (10−410^{-4}), rank (r=3r=3), and polar reconstruction. The class-mean variant averages the gradients within each class before truncation.

At 70 support-set passes, per-instance merging improves accuracy on all four datasets at both shot counts. The gains average 1.25 and 2.55 percentage points at 4 and 8 shots, respectively. To match the number of optimizer updates, class-mean merging requires 280 and 560 passes. At 4 shots, per-instance and class-mean merging then reach 66.12% and 66.19%. At 8 shots, they reach 70.94% and 71.42%.

For example, on 4-shot FGVC-Aircraft, both matched schedules perform 560 updates and decompose 28,000 gradient matrices per eligible tensor. Per-instance merging processes 28,000 example presentations, compared with 112,000 for class-mean merging.

AuthorsOpenReview ↗

Response to Reviewer 9vwD (2/3)

Response to Reviewer 9vwD (2/3)

C3. Computational cost and practical efficiency

We profiled OLM-SGD (r=3r=3), SGD, and rank-2 LoRA [1] trained with Adam on SUN397 using CLIP ViT-B/16. All runs used the same NVIDIA H100 GPU, seed 1, a fixed learning rate of 10−410^{-4}, and K∈1,2,4K\in\\{1,2,4\\}.

Training cost and GPU memory. At one shot, an OLM-SGD support pass takes 89.06 s versus 2.28 s for SGD, approximately 39 times as long. Both methods update 122.88 million parameters. Peak allocation is 15.30 GiB for OLM-SGD, 0.91 GiB above SGD and 0.14 GiB above LoRA (Table 1).

Table 1. Training cost and peak allocated memory (K=1K=1, one H100 GPU).

Method         Trainable (M)  Peak memory (GiB)  Time/pass (s)
SGD                   122.88              14.39           2.28
LoRA (rank 2)           0.55              15.16           2.47
OLM-SGD (r=3)         122.88              15.30          89.06

In one instrumented 64-example OLM-SGD update, individual backward passes take 4.89 s, truncated SVDs take 8.31 s, and factor assembly and polar decomposition take 1.46 s. The truncated SVDs account for 54% of the instrumented update time.

Accuracy under matched wall-clock budgets. We fixed the budgets of 667, 1,334, and 2,668 s before evaluating accuracy. Checkpoints were selected by elapsed training time. SGD leads at all three shot counts at 667 s. At 1,334 s, OLM-SGD leads at one shot and SGD leads at two and four shots. At 2,668 s, OLM-SGD achieves the highest accuracy at all three shot counts (Table 2). Its gains over LoRA are 5.16, 4.10, and 3.47 percentage points.

Table 2. SUN397 test accuracy (%) at a matched training-time budget of 2,668 s.

Shots (K)    SGD  LoRA (rank 2)  OLM-SGD (r=3)  Gain over SGD
1          69.10          64.94         70.10           +1.00
2          70.85          67.31         71.41           +0.56
4          72.11          68.92         72.39           +0.28

Appendix J reports the actual training time and accuracy of all nine method–shot combinations at each budget. These measurements use one seed.

[1] Low-Rank Few-Shot Adaptation of Vision-Language Models, Zanella and Ben Ayed, CVPRW 2024.

AuthorsOpenReview ↗

Response to Reviewer 9vwD (3/3)

Q1. Generalization beyond few-shot adaptation

We evaluated few-shot adaptation of pretrained CLIP across 11 datasets, three backbones, and one to 16 labeled examples per class. Whether OLM-SGD improves standard full-data fine-tuning or pretraining remains an open empirical question. The current implementation computes a separate gradient for each example before merging. On SUN397 with ViT-B/16, increasing the support set from one to four shots raises OLM-SGD training time from 89.06 to 348.04 s per pass at the same batch capacity (Appendix J). This cost is particularly relevant when moving to larger training sets.

Q2. Applying OLM-SGD to LoRA

OLM-SGD can train the LoRA factors of both CLIP encoders. Rank-2 LoRA uses 0.553 million trainable parameters, a 99.55% reduction relative to the 122.88 million updated in full fine-tuning. In the tested one-shot setting, the gain observed in full fine-tuning did not carry over to LoRA. Mean accuracy was 44.09% for LoRA-OLM-SGD, 44.39% for LoRA-SGD, and 52.53% for LoRA-Adam.

We compared the three optimizers on EuroSAT, FGVC-Aircraft, Stanford Cars, and DTD with ViT-B/16. All use the same rank-2 adapters, initialization, support sets, and minibatch orders. Each run uses one shot per class, a learning rate of 10−410^{-4}, and 100 epochs. We average over three seeds. OLM-SGD uses nominal rank r=3r=3.

Table 3. Test accuracy (%) and training loss at epoch 100 for rank-2 LoRA, with best-to-final test accuracy degradation and peak epoch.

Optimizer     EuroSAT  Aircraft   Cars    DTD  Mean acc.
LoRA-Adam       67.76     25.50  64.33  52.52     52.53
LoRA-SGD        43.08     24.25  65.69  44.54      44.39
LoRA-OLM-SGD    42.45     23.95  65.56  44.41      44.09

Optimizer     Train loss ↓  Drop (pp) ↓  Peak ep.
LoRA-Adam          0.0013          2.24        38
LoRA-SGD            1.9172         0.02        98
LoRA-OLM-SGD        1.9451        0.01         92

The accuracy and training loss are averaged over three seeds and four datasets. The best-to-final drop is averaged over all 12 individual trajectories per optimizer. Peak epoch is determined by the macro-averaged test accuracy across datasets and seeds.

Optimization at the common endpoint. At epoch 100, mean training accuracy is 42.55% for LoRA-OLM-SGD, 42.69% for LoRA-SGD, and 100.00% for LoRA-Adam. The two SGD-based variants still have mean training losses above 1.9. Their small peak-to-final test drops therefore occur while the adapters are still fitting the support set. The experiment establishes that OLM-SGD can optimize rank-2 adapters, with the strongest mean accuracy under this schedule obtained by LoRA-Adam.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.