- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the problem of transferring knowledge to a target task with limited labeled data, given only a set of source checkpoints or adapters fine-tuned from a common pre-trained model, without access to the original source data. To define transfer in the function space of target inputs rather than in parameter space, the authors map each source update to a change in target logits via the base model’s Jacobian and compare it against a descent direction adjusted by the curvature of the target loss. Based on this, they define the principal transfer angle, which represents the utility of each source, and transfer completeness, which indicates the extent to which the source library covers the target adaptation. The proposed method, GeoTransfer, determines combination coefficients by solving an orthogonal projection problem with sparse-group regularization over layer-grouped source updates, learns the components unexplained by the sources using an adapter penalized for overlap with the selected source directions, and employs a risk gate that permits deployment only if a statistical test on validation data confirms that performance is not worse than a target-only model. Theoretically, the paper establishes projection optimality under a second-order approximation, first-order invariance to re-parameterization, an upper bound on the sparse estimation error that depends on the number of active sources, and a finite-sample guarantee for the risk gate. Experiments spanning controlled simulations, CLIP task libraries, cross-domain vision libraries, and language adapter libraries report better source ranking and few-shot performance than the baselines, along with a substantial reduction in negative transfer.
Critical strengths
Principled formulation and interpretable diagnostics. By defining source utility in function space rather than parameter space, the authors provide a criterion that is robust to parameter symmetries such as rescaling, and connect it to a clear geometric structure based on orthogonal projection. In particular, the completeness metric provides information that existing transferability scores or merging methods do not offer, since it quantifies not only the ranking of individual sources but also how much of the target adaptation the entire library can cover. The theoretical framework is also rigorous, with explicit assumptions and proofs, and a strict separation of the fit, validation, and test splits.
Practical safeguards and computational efficiency. The method includes a validation-based risk gate intended to control negative transfer, backed by a finite-sample guarantee. Furthermore, the required quantities are computed with Jacobian-vector products and a single backward pass, without constructing a parameter-dimensional Hessian, and at deployment the model runs at the same cost as a standard model, with no additional source models or routers.
Critical weaknesses
The positioning relative to related work and the choice of baselines are inadequate. The setting of adapting to a new target in a few-shot manner by combining task vectors fine-tuned from a pre-trained CLIP model with block-wise coefficients was already studied in aTLAS (Zhang et al., NeurIPS 2024) under nearly the same protocol (leave-one-out task vectors, k shots per class), yet this paper neither cites nor compares against it. There is also no comparison with LoraHub, which directly overlaps with the language experiments, or with AXIS (Osial et al., 2025), which the paper itself cites as the closest related work. In addition, Ortiz-Jiménez et al. (NeurIPS 2023), which studies task vectors in the tangent space and is closely related to the first-order view adopted here, is not cited or discussed. The current baselines consist mainly of methods originally designed for multi-task merging (e.g., TIES, TATR, TSV-Merge), which make little use of target labels, together with author-constructed baselines, and standard CLIP few-shot baselines such as Tip-Adapter and LP++ are absent. As a result, the remaining core contribution, namely determining coefficients through orthogonal projection in function space, is compared against direct optimization of coefficients on the target loss only through an author-constructed baseline whose granularity and regularization are not specified, and not against established methods such as aTLAS.
The empirical evidence on real data is weak. Table 2 does not report standard errors, the number of random seeds, or per-dataset results. Information needed for reproducibility is also missing, including the backbone, the number of LoRA experts, and the target tasks in the language experiments, as well as the block partitioning criterion in the vision experiments. In addition, the differences between methods in Table 2 are unusually constant across six very different settings; for example, applying the gate lowers performance by exactly 0.2 percentage points in every setting. The accuracy of the first-order (tangent) approximation, on which the method relies, is not measured on real models, even though real CLIP task vectors can lie far from the base model. Meanwhile, the simulation favors the proposed method, since the sources are generated by orthogonalizing latent factors under the same function-space inner product that the method uses. The source-ranking criterion, the gain from adding a single source at its optimal scale, is essentially the quantity that the proposed angle approximates, which makes the comparison with methods such as LogME questionable. Finally, the claim that the completeness metric predicts actual transfer gains is supported only by simulation results.
Questions for authors
- Could you provide comparisons with aTLAS, LoraHub, and AXIS under the same conditions? In addition, please specify whether the Sparse target-loss weights baseline operates at the source level or the block level, and whether it uses the same atom partitioning as the proposed method.
- Could you report the number of seeds, per-seed results with standard errors, and per-dataset results for Table 2 and Table F.3? Please also specify the backbone, the LoRA expert configuration, and the target tasks for the language experiments, as well as the criterion for the three-group block partitioning in the vision experiments.
Final Assessment
Thank you for your response. I still see the paper as borderline between 2 and 3, and I will keep my rating at 3.