We thank the reviewer for the careful assessment. We address the concerns below.
C1. Novelty relative to ATM and TSV-M
We thank the reviewer for highlighting ATM (Zhou et al., 2024) and On Task Vectors and Gradients (Zhou et al., 2025), which primarily investigate task-vector merging across multiple tasks. Our work addresses a distinct question: can a single few-shot classification task benefit from multitask-inspired merging of class-specific gradients? To our knowledge, OLM-SGD is the first approach to investigate this perspective for few-shot CLIP adaptation. Unlike multitask model merging, we consider one classification task on a single dataset, where each instance gradient is computed using the full multiclass loss.
We study three questions:
- Gradient aggregation: Can preserving distinct class-specific instance gradients through orthogonal low-rank merging improve discriminative adaptation compared with conventional gradient averaging?
- Optimization mechanism: Does modifying the gradient direction through polar reconstruction provide benefits beyond low-rank truncation and update-norm control?
- Optimization robustness: Can structured gradient merging reduce sensitivity to learning rates and training duration, thereby enabling reliable few-shot adaptation under validation-free protocols?
Algorithmically, OLM-SGD combines class-unique minibatching with tensor-wise spectral merging of individual gradients at each optimization step. It directly adapts both CLIP encoders without additional trainable modules or adaptive optimizer states. We acknowledge the mathematical connection to TSV-M with an ATM-style construction with singleton subtasks and one-step updates. However, neither TSV-M nor ATM investigate class-unique gradient merging within a single multiclass objective or its implications for few-shot adaptation robustness.
Theoretically, we establish retained-gradient alignment and energy bounds, conditional stationarity guarantees, and finite-shot population bounds accounting for repeated support-set use. These results characterize the spectral merging mechanism within a stochastic optimization framework rather than as a post-training model-merging operation.
Empirically, across 11 datasets, three CLIP backbones, and five shot settings, OLM-SGD demonstrates accuracy improvements over the evaluated baselines. Controlled low-rank truncation and polar-reconstruction ablations, together with direction–norm interventions, establish the importance of the proposed gradient-merging mechanism. Beyond accuracy, extensive learning-rate sweeps, 300-epoch training trajectories, and cross-dataset epoch-transfer experiments demonstrate improved learning-rate robustness, long-horizon stability, and reduced sensitivity to training duration. These properties are particularly valuable for validation-free few-shot adaptation, where reliable hyperparameter selection and early stopping are challenging.
C2. Full fine-tuning SGD baseline and matched training budgets
We agree that full-parameter SGD is the primary optimizer baseline. The existing comparison in Appendix H.1 uses the same trainable CLIP parameters, classification objective, and learning rate (10−4), across 11 datasets, three backbones, five shot levels, and three seeds. OLM-SGD leads SGD in 14 of the 15 backbone–shot averages. Its gains are 0.3–0.9 points on ViT-B/32 and 0.5–1.2 points on ViT-B/16. On ViT-L/14, it leads by 0.1–0.5 points at 2–16 shots. The revised abstract reports the comparison with SGD. Table 1 of the manuscript includes SGD across all 15 settings.
Why 30 epochs for SGD and 70 for OLM-SGD? The original evaluation applied these method-specific schedules unchanged across datasets, backbones, and shot levels, rather than selecting a separate stopping epoch for every benchmark. Table 2 of the manuscript compares the fixed schedules with the peaks of the 300-epoch one-shot trajectories. At 10−4, the dataset-average SGD curve peaks at epoch 21 (71.61%), while its fixed epoch-30 accuracy is 71.49%, only 0.12 points below the peak. OLM-SGD peaks at epoch 67 (72.05%), while its epoch-70 accuracy is 72.02%, only 0.03 points below the peak. Even at their respective retrospective peaks, OLM-SGD retains a 0.44-point advantage.
Table 4. Original schedules and retrospective long-horizon peaks (1-shot ViT-B/16, 11 datasets, seeds 1–3, η=10−4). Accuracy is in percent; gaps and best-to-final drops are percentage points.
Method Fixed epoch/acc. Peak epoch/acc. Gap E300 acc. Drop
SGD 30 / 71.49 21 / 71.61 0.12 70.46 1.81
OLM-SGD 70 / 72.02 67 / 72.05 0.03 71.75 0.95
Response to Reviewer B3C
We thank the reviewer for this constructive comment.
C1. Matched training budgets and the class-mean control
Matched updates with SGD. Appendix H.2 compares OLM-SGD and class-unique SGD on the same trainable parameters at η=10−4. Both methods match scheduled updates and example counts. Across 11 one-shot datasets on ViT-B/16, OLM-SGD improves mean test accuracy by 0.13, 0.83, and 1.31 percentage points at epochs 30, 70, and 300, respectively. At 16 shots and 70 epochs, OLM-SGD reaches 76.50% across EuroSAT, FGVC-Aircraft, and Stanford Cars, compared with 74.85% for class-unique SGD and 74.74% for random-minibatch SGD. These results average three seeds.
Matched updates with class-mean merging. Appendix H.5 compares the variants on four datasets with ViT-B/16, r=3, and three seeds. At equal support-set exposure (70 passes), per-example merging improves mean accuracy by 1.25 points at 4 shots and 2.55 points at 8 shots. At equal update counts, per-example and class-mean merging reach 66.12% and 66.19% at 4 shots, and 70.94% and 71.42% at 8 shots. Per-example merging uses 70 passes. Class-mean merging uses 280 and 560, respectively.
Matched backward passes. Appendix H.10 isolates the polar step. Both variants compute and truncate the same number of per-example gradients per update. Replacing the arithmetic mean of the truncated gradients with joint polar reconstruction improves accuracy in all 12 dataset–shot settings. At 16 shots, mean accuracy increases from 74.31% to 75.54%.
C2. Training time, memory, and computational scaling
Appendix J profiles SUN397 on one NVIDIA H100 with ViT-B/16 and batch capacity 64. At one shot, OLM-SGD, SGD, and rank-2 LoRA take 89.06, 2.28, and 2.47 seconds per support pass. Peak allocations are 15.30, 14.39, and 15.16 GiB, respectively. OLM-SGD takes 176.24 seconds per pass at two shots and 348.04 at four. Its peak allocation stays at 15.30 GiB. This measures scaling with support size at fixed class count, batch capacity, and backbone.
Cost per update. In one 64-example update, truncated SVDs take 8.31 seconds (54% of update time), per-example backward passes take 4.89 seconds, and factor assembly and polar decomposition take 1.46 seconds. Appendix C.1 counts CK gradient decompositions and 2K⌈C/B⌉ polar decompositions per connected trainable tensor per pass. For a tensor of shape mW×nW, the concatenated factors have mW and nW rows and Rt=rWBt columns. When Rt≤min(mW,nW), the two reduced SVDs in the polar step cost O((mW+nW)Rt2) operations per tensor update. The end-to-end profile reports time and peak memory. End-to-end FLOPs and cross-backbone timings remain unmeasured.
Accuracy at equal training time. SGD leads at all three shot counts at 667 seconds. At 2,668 seconds, OLM-SGD leads SGD by 1.00, 0.56, and 0.28 percentage points at one, two, and four shots, respectively. Its gains over LoRA are 5.16, 4.10, and 3.47 points. Both budgets were fixed before evaluation (SUN397, seed 1).
Q1. Alignment and reconstruction along training trajectories
Appendix H.11 measures alignment with ∇F(Θt) computed over the entire one-shot support set. For EuroSAT, FGVC-Aircraft, and Stanford Cars on ViT-B/16, the 36 initial configurations share nine dataset–seed starts across four rates. We also probed 52 trained checkpoints from 19 trajectories. The update gt has positive inner product with the full gradient on 447 of 448 minibatches, including 279 of 280 at trained checkpoints. At epoch 300 and η=10−5, the mean cosines on full-size minibatches are 0.201 on FGVC-Aircraft and 0.179 on Stanford Cars across three seeds.
Reconstruction errors. On Stanford Cars at η=10−4 and Bt=64, we normalize each error by ∣∇F(Θt)∣2 before averaging over minibatches and seeds. At initialization and epoch 300, the normalized truncation bound εt,tail is 5.058 and 4.668, respectively. The normalized polar reconstruction error εt,polar is 1.562 and 1.549. The absolute batch merge error ∣gt−gt∣2 falls from 55.93 to 0.819.
Theoretical conditions and observed descent. All 448 minibatches satisfy the retained-alignment and energy inequalities in Lemma 4.1. Both sufficient lower bounds for full-gradient alignment are nonpositive on all draws. The measured inner products describe individual draws. The stationarity assumptions concern conditional expectations over draws at fixed parameters. Finite-step trials begin at identical parameters. At η=10−3 and the same displacement norm, the merged direction reduces the full support loss at all nine initial states. The raw and truncated means increase it at all nine.