- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes REFRACT, a model re-basin framework for transferring task-specific capabilities between pre-trained models with different architectures or initializations. The method leverages linearized fine-tuning in the neural tangent kernel (NTK) regime to decompose the source model's functional residual into additive block-wise contributions. These residuals are first mapped into the target model's feature space, after which lightweight linear predictors are trained to reconstruct the transferred corrections from the frozen target model's intermediate activations. Experiments on vision classification, natural language inference, and mathematical reasoning tasks demonstrate improvements over several existing transfer methods, without requiring source-model inference or fine-tuning of the target backbone.
Critical strengths
- Solid mathematical formulation.
- Broad experimental evaluation.
Critical weaknesses
-
Limited Methodological Novelty. The proposed method primarily combines existing techniques, including NTK-based linearized fine-tuning, linear regression for residual transport, and ridge regression for block-wise prediction. In particular, the exact additive decomposition in Eq. (4) follows directly from the linearity of the Jacobian-vector product, rather than representing a fundamentally new theoretical contribution.
-
Insufficient Validation of the Core Contribution. The experiments do not sufficiently isolate the benefits of NTK-based block-wise decomposition, as REFRACT uses multiple intermediate target activations while the global predictor baselines primarily rely on final-layer features. Therefore, it remains unclear whether the improvements arise from the proposed decomposition or simply from richer intermediate representations. A controlled comparison using identical target features would be necessary.
-
Limited Baseline Comparisons. The cross-architecture evaluation in Table 1 includes only two external methods, Theseus and BiCo, with the latter cited as an arXiv preprint. More importantly, the paper lacks comparisons with simpler alternatives, such as directly predicting the teacher–target logit residual from frozen target activations or performing target-only few-shot adaptation. These baselines are important for establishing whether the proposed mechanism offers meaningful advantages over standard supervised residual prediction.
Questions for authors
-
Could the authors clarify what constitutes the key methodological novelty beyond the combination of existing NTK-based linearization and standard regression techniques?
-
Could the authors include additional baselines, particularly direct teacher–target logit residual prediction and target-only few-shot adaptation under the same calibration budget, to better establish the advantages of REFRACT?
5 comments
Direct teacher–target logit residual prediction
More importantly, the paper lacks comparisons with simpler alternatives, such as directly predicting the teacher–target logit residual from frozen target activations [...]. These baselines are important for establishing whether the proposed mechanism offers meaningful advantages over standard supervised residual prediction.
Following the reviewer's suggestion, we evaluated a direct teacher–target logit residual predictor: a ridge regressor on the frozen target features predicts , which is added to the target logits at inference:
Like ReFract, it is fitted in closed form and does not require the source model at inference; with , it coincides with the distillation baseline in our response to Rev. 8wHE.
| Setting | Method | K=1 | K=2 | K=5 | K=10 | K=20 |
|---|---|---|---|---|---|---|
| ViT-B/16 → ViT-B/16 | Logit residual prediction | 68.28 | 74.44 | 77.19 | 80.26 | 82.51 |
| ReFract | 72.20 | 77.52 | 81.84 | 83.97 | 85.55 | |
| ViT-B/16 → ViT-L/14 | Logit residual prediction | 75.86 | 79.16 | 83.54 | 85.69 | 86.93 |
| ReFract | 80.55 | 83.72 | 86.41 | 87.56 | 88.72 |
ReFract consistently outperforms this baseline (+3.0 to +4.7 points on ViT-B/16 → ViT-B/16 and +1.8 to +4.7 on ViT-B/16 → ViT-L/14). Together with ReFract-Joint, this shows that learning block-specific targets from the decomposed donor residual offers a meaningful advantage over directly regressing the logit residual.
Target-only few-shot adaptation
More importantly, the paper lacks comparisons with simpler alternatives, such as directly predicting the teacher–target logit residual from frozen target activations or performing target-only few-shot adaptation. These baselines are important for establishing whether the proposed mechanism offers meaningful advantages over standard supervised residual prediction.
Also, in the Questions section:
Could the authors include additional baselines, particularly direct teacher–target logit residual prediction and target-only few-shot adaptation under the same calibration budget, to better establish the advantages of REFRACT?
We thank the reviewer for this suggestion, which was also raised by Rev. 8wHE and Rev. kGVk. Following it, we ran an experiment in which all methods share the same data budget, i.e., the same number of examples per class for each task. We evaluate full fine-tuning, LoRA fine-tuning and knowledge distillation (with full fine-tuning of the target model). We refer the reviewer to our response to Rev. 8wHE ("Comparison with direct receiver adaptation") for the complete discussion, and report the results below for convenience.
| Setting | Method | K=1 | K=2 | K=5 | K=10 | K=20 |
|---|---|---|---|---|---|---|
| ViT-B/16 → ViT-B/16 | Full FT | 74.61 | 81.53 | 84.95 | 87.33 | 88.60 |
| LoRA | 74.96 | 79.46 | 84.06 | 86.58 | 88.87 | |
| KD (Full FT) | 73.80 | 79.11 | 84.34 | 87.53 | 89.73 | |
| ReFract | 72.20 | 77.52 | 81.84 | 83.97 | 85.55 | |
| ViT-B/16 → ViT-L/14 | Full FT | 77.98 | 82.80 | 87.13 | 89.78 | 91.00 |
| LoRA | 78.13 | 80.42 | 86.01 | 89.03 | 91.33 | |
| KD (Full FT) | 79.68 | 82.61 | 87.70 | 90.06 | 91.87 | |
| ReFract | 80.55 | 83.72 | 86.41 | 87.56 | 88.72 |
In short, methods that adapt the target backbone through multi-step gradient descent generally achieve higher accuracy as the data budget grows: in the same-architecture setting, they outperform ReFract at every budget (by 1.6 to 4.2 points; the gap narrows to at most 2.2 points when excluding SVHN, as discussed in our response to Rev. 8wHE), while in the cross-architecture setting full fine-tuning and KD surpass ReFract only from K=5 onward, and LoRA only from K=10. In the few-shot regime, instead, ReFract surpasses full fine-tuning, LoRA, and KD at K=1 and K=2 in the cross-architecture setting (80.55 vs. 77.98, 78.13, and 79.68 at K=1; 83.72 vs. 82.80, 80.42, and 82.61 at K=2), through closed-form fitting and without backpropagation through the target backbone.
We thank you again for your constructive feedback. As the discussion period is coming to an end, we hope that our clarifications on the methodological novelty, controlled ablations isolating the benefits of decomposition, and additional comparisons with simpler baselines have addressed your main concerns. Please let us know if anything remains unclear. We sincerely appreciate your time and consideration.
We thank Reviewer DUug for their constructive feedback. Below, we clarify our contribution and provide additional controlled comparisons and baselines.
Methodological novelty
Also, in the Questions section:
We thank the reviewer for the opportunity to clarify our methodological contribution. We agree that Eq. (4) follows directly from the linearity of the Jacobian-vector product, and we do not claim it as a new theoretical result. Our contribution lies in using this property to construct a structured transfer procedure across potentially heterogeneous pretrained models, with three main aspects:
From weight-space updates to activation-space residuals. We shift the focus of fine-tuning transfer from parameter updates to their functional effects in activation space. This avoids requiring direct parameter correspondence and provides greater flexibility when source and target architectures differ substantially.
A mechanistic view of transfer through decomposed adaptation. We decompose the global functional update into contributions associated with individual parameter blocks, transport them separately, and use them to supervise predictors on the receiver. Tangent-space fine-tuning makes this decomposition exact, providing a principled basis for breaking a global transfer problem into smaller, structured prediction problems.
Fine-tuning that is rebasable by design. We investigate whether the source adaptation procedure itself can produce functional updates that are easier to transfer across backbones. This brings transferability into the choice of how the source is fine-tuned, rather than treating transfer solely as a post-hoc problem. Tangent-space fine-tuning offers a concrete realization of this perspective through its exact additive structure.
Our contribution is therefore methodological and empirical: using functional residuals and their decomposition to enable few-shot transfer, and evaluating the benefits of this design against global prediction and direct adaptation.
Following the reviewer's comment, we have added a paragraph summarizing these contributions at the end of the Related Work section (Sec. 6, "Summary of contributions") in the revised manuscript.
We hope that this clarifies the novelty of our contribution, which, incidentally, Rev. kGVk explicitly listed among the strengths of our work, describing our approach as "innovative" and noting that "the proposed shift from parameter-space alignment to NTK-based functional residual decomposition offers an interesting and theoretically motivated perspective."
Validation of the core contribution
Controlled comparison with identical features and capacity
We thank the reviewer for raising this point. To address it, we ran a controlled ablation (ReFract-Joint): the multi-layer receiver features are concatenated into a single vector, and a single ridge regressor predicts the global residual transported with the same Stage 1. ReFract-Joint thus uses the same features and number of parameters as ReFract, and differs only in whether the residual is decomposed into block-wise contributions.
As shown, there is a clear gap between the two methods (e.g., 3.4–4.6 points at K ≥ 2), indicating that the advantage of our method stems not only from richer features but also from its block-specific targets. We have added this comparison to the main paper (Sec. 5, "Joint vs. decomposed prediction", Fig. 4b, covering both vision settings), with complete results in App. C.6.