- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
In the work the authors introduce Cayley-Parameterized Task-Vector Transfer (CTT), a method for transferring fine-tuning updates across pretrained models with the same architecture by learning orthogonal residual-stream rotations jointly with their scaling coefficients. The Cayley transform is used to maintain orthogonality diring optimization, with the motivation that the resulting transformation respects reparameterization symmetries and thus changes the basis of transferred update rather than its content. The method is evaluated on CLIP image and text tasks and on natural-language autoencoders.
Critical strengths
CTT has a clean motivation, the transferred task vector is transformed only through an architectural reparameterization symmetry, with the Cayley parameterization maintaining orthogonality throughout optimization.
The paper evaluates the method on CLIP image/text transfer and on natural-language autoencoders, and compares against several recent task-vector transfer methods.
Critical weaknesses
-
The experimental protocol is not sufficiently described in the main paper. Essential information needed to interpret the results are buried in the very long supplementary material without any pointer. For example, the definitions of TASK/REC/SUPP/RET are missing and are different from usual settings, the most similar is TransFusion which reports TASK and SUPP. The few-shot budget, optimization details and validation protocol are all hidden in the appendix. The most critial is the averaging procedure of all budgets at different seeds which should be clearly disclosed in the main paper. All these lack of details make it impossibile to interpret the results without reading the 30+ pages appendix.
-
The CLIP experiments do not isolate transfer from supervised adaptation. CTT is not a label-free transport procedure. It takes the labeled examples from the target task and explicitly optimize R and by cross-entropy on those labels for 500 updates (this is unclear, but seems the case from the appendix). At this point the question is obvious: why not simply adapt the target model using such samples. Without this comparison it is hard to know if CTT is exploiting knowledge in the source task vector or simply is a constrained parameterization for few shot supervised adaptation. From the original CLIP paper we know that 16 shots is already more than enough to fit the model on the downstream task. Note that GradFix and BiCo make use of the gradients, but do not directly use gradient descent, moreover, GradFix reports a same budget target finetuning comparison.
-
Missing baseline. CTT is initialized from a validation-selected task vector scale, but this tuned naive-transfer baseline is not reported. A tuned-scale or -only baseline is needed to separate the gain from scale selection from the gain due to the Cayley rotation.
-
The comparison across transfer methods is not budget-matched. Different methods use substantially different amounts and types of information: some are data-free, some use unlabeled data, while CTT uses labeled examples, gradient-based optimization, and repeated validation-based checkpoint selection. These differences are mostly hidden in the supplement and complicate claims of superiority.
-
The empirical evidence for generality is limited. The CLIP experiments consider a single source/recipient checkpoint pair with the same ViT-B/16 architecture, and support retention is evaluated rather narrowly. This seems limited relative to the broader framing of task-vector transfer across pretrained models.
Questions for authors
-
Can you compare CTT against direct adaptation of the recipient CLIP model using exactly the same labeled examples and a matched optimization budget, including full fine-tuning and/or LoRA?
-
Can you report the validation-tuned naive-transfer baseline used to initialize CTT, together with an -only learned baseline, so that the contribution of the Cayley rotation can be isolated?
-
Can you provide a matched accounting of the labeled/unlabeled data, optimization steps or FLOPs, and number of validation evaluations available to each method?
-
Can the essential experimental protocol currently buried in the supplement, including the definitions of TASK/REC/SUPP/RET, the -shot budgets and averaging procedure, checkpoint pair, and model-selection protocol, be moved into the main paper? In the current version, I do not think the central tables are self-contained enough to assess the claims without substantial appendix reading.