acceptodds
Under review as a conference paper at ICLR 2027

ReFract: Model Re-basin through Decomposable Fine-Tuning Residuals

13 views

Abstract

Model re-basin enables the transfer of task-specific capabilities between models, mitigating the challenges of full retraining. However, existing parameter-space approaches typically rely on weight correspondences that are difficult to establish across different architectures. To address this limitation, we shift the focus to activation space and transfer the functional residual, i.e., the change in last-layer hidden representations induced by fine-tuning. Under standard non-linear fine-tuning, this residual entangles the contributions of individual parameter updates through non-linear interactions, making them difficult to isolate and transport to another model. We show that, when the donor model is fine-tuned in the neural tangent kernel (NTK) regime, the residual admits an exact additive decomposition over parameters, without higher-order interactions. Like a prism separating light into its constituent components, we isolate the functional contribution of each layer and transport it to corresponding layers in another model. In practice, our approach first transports these activation residuals into the target model's feature space and then learns layer-wise predictors that map its frozen intermediate activations to the transported corrections. We achieve state-of-the-art performance in capability transfer across both vision and language models, approaching full-data target fine-tuning using only a few examples from the original dataset.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
model re-basin, model editing, neural tangent kernel, transfer learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DUugOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes REFRACT, a model re-basin framework for transferring task-specific capabilities between pre-trained models with different architectures or initializations. The method leverages linearized fine-tuning in the neural tangent kernel (NTK) regime to decompose the source model's functional residual into additive block-wise contributions. These residuals are first mapped into the target model's feature space, after which lightweight linear predictors are trained to reconstruct the transferred corrections from the frozen target model's intermediate activations. Experiments on vision classification, natural language inference, and mathematical reasoning tasks demonstrate improvements over several existing transfer methods, without requiring source-model inference or fine-tuning of the target backbone.

Critical strengths
  1. Solid mathematical formulation.
  2. Broad experimental evaluation.
Critical weaknesses
  1. Limited Methodological Novelty. The proposed method primarily combines existing techniques, including NTK-based linearized fine-tuning, linear regression for residual transport, and ridge regression for block-wise prediction. In particular, the exact additive decomposition in Eq. (4) follows directly from the linearity of the Jacobian-vector product, rather than representing a fundamentally new theoretical contribution.

  2. Insufficient Validation of the Core Contribution. The experiments do not sufficiently isolate the benefits of NTK-based block-wise decomposition, as REFRACT uses multiple intermediate target activations while the global predictor baselines primarily rely on final-layer features. Therefore, it remains unclear whether the improvements arise from the proposed decomposition or simply from richer intermediate representations. A controlled comparison using identical target features would be necessary.

  3. Limited Baseline Comparisons. The cross-architecture evaluation in Table 1 includes only two external methods, Theseus and BiCo, with the latter cited as an arXiv preprint. More importantly, the paper lacks comparisons with simpler alternatives, such as directly predicting the teacher–target logit residual from frozen target activations or performing target-only few-shot adaptation. These baselines are important for establishing whether the proposed mechanism offers meaningful advantages over standard supervised residual prediction.

Questions for authors
  1. Could the authors clarify what constitutes the key methodological novelty beyond the combination of existing NTK-based linearization and standard regression techniques?

  2. Could the authors include additional baselines, particularly direct teacher–target logit residual prediction and target-only few-shot adaptation under the same calibration budget, to better establish the advantages of REFRACT?

5 comments

We thank Reviewer DUug for their constructive feedback. Below, we clarify our contribution and provide additional controlled comparisons and baselines.

Methodological novelty

The proposed method primarily combines existing techniques, including NTK-based linearized fine-tuning, linear regression for residual transport, and ridge regression for block-wise prediction. In particular, the exact additive decomposition in Eq. (4) follows directly from the linearity of the Jacobian-vector product, rather than representing a fundamentally new theoretical contribution.

Also, in the Questions section:

Could the authors clarify what constitutes the key methodological novelty beyond the combination of existing NTK-based linearization and standard regression techniques?

We thank the reviewer for the opportunity to clarify our methodological contribution. We agree that Eq. (4) follows directly from the linearity of the Jacobian-vector product, and we do not claim it as a new theoretical result. Our contribution lies in using this property to construct a structured transfer procedure across potentially heterogeneous pretrained models, with three main aspects:

  1. From weight-space updates to activation-space residuals. We shift the focus of fine-tuning transfer from parameter updates to their functional effects in activation space. This avoids requiring direct parameter correspondence and provides greater flexibility when source and target architectures differ substantially.

  2. A mechanistic view of transfer through decomposed adaptation. We decompose the global functional update into contributions associated with individual parameter blocks, transport them separately, and use them to supervise predictors on the receiver. Tangent-space fine-tuning makes this decomposition exact, providing a principled basis for breaking a global transfer problem into smaller, structured prediction problems.

  3. Fine-tuning that is rebasable by design. We investigate whether the source adaptation procedure itself can produce functional updates that are easier to transfer across backbones. This brings transferability into the choice of how the source is fine-tuned, rather than treating transfer solely as a post-hoc problem. Tangent-space fine-tuning offers a concrete realization of this perspective through its exact additive structure.

Our contribution is therefore methodological and empirical: using functional residuals and their decomposition to enable few-shot transfer, and evaluating the benefits of this design against global prediction and direct adaptation.

Following the reviewer's comment, we have added a paragraph summarizing these contributions at the end of the Related Work section (Sec. 6, "Summary of contributions") in the revised manuscript.

We hope that this clarifies the novelty of our contribution, which, incidentally, Rev. kGVk explicitly listed among the strengths of our work, describing our approach as "innovative" and noting that "the proposed shift from parameter-space alignment to NTK-based functional residual decomposition offers an interesting and theoretically motivated perspective."

Validation of the core contribution

The experiments do not sufficiently isolate the benefits of NTK-based block-wise decomposition, as REFRACT uses multiple intermediate target activations while the global predictor baselines primarily rely on final-layer features. Therefore, it remains unclear whether the improvements arise from the proposed decomposition or simply from richer intermediate representations. A controlled comparison using identical target features would be necessary.

Controlled comparison with identical features and capacity

We thank the reviewer for raising this point. To address it, we ran a controlled ablation (ReFract-Joint): the multi-layer receiver features are concatenated into a single vector, and a single ridge regressor predicts the global residual transported with the same Stage 1. ReFract-Joint thus uses the same features and number of parameters as ReFract, and differs only in whether the residual is decomposed into block-wise contributions.

SettingMethodK=1K=2K=5K=10K=20
ViT-B/16 → ViT-B/16ReFract-Joint71.1073.5077.2079.4081.00
ReFract72.2077.5281.8483.9785.55
ViT-B/16 → ViT-L/14ReFract-Joint78.0980.1082.4683.9485.27
ReFract80.5583.7286.4187.5688.72

As shown, there is a clear gap between the two methods (e.g., 3.4–4.6 points at K ≥ 2), indicating that the advantage of our method stems not only from richer features but also from its block-specific targets. We have added this comparison to the main paper (Sec. 5, "Joint vs. decomposed prediction", Fig. 4b, covering both vision settings), with complete results in App. C.6.

Direct teacher–target logit residual prediction

More importantly, the paper lacks comparisons with simpler alternatives, such as directly predicting the teacher–target logit residual from frozen target activations [...]. These baselines are important for establishing whether the proposed mechanism offers meaningful advantages over standard supervised residual prediction.

Following the reviewer's suggestion, we evaluated a direct teacher–target logit residual predictor: a ridge regressor on the frozen target features HB(x)H_B(x) predicts zAft(x)−zB0(x)z_A^{ft}(x) - z_B^0(x), which is added to the target logits at inference:

min⁡Δ ∣∣(zAft(x)−zB0(x))−ΔHB(x)∣∣2+λ∣∣Δ∣∣2.\min_{\Delta}\ || \big(z_A^{ft}(x) - z_B^0(x)\big) - \Delta H_B(x) ||^2 + \lambda || \Delta ||^2 .

Like ReFract, it is fitted in closed form and does not require the source model at inference; with TB=TB0+ΔT_B = T_B^0 + \Delta, it coincides with the distillation baseline in our response to Rev. 8wHE.

SettingMethodK=1K=2K=5K=10K=20
ViT-B/16 → ViT-B/16Logit residual prediction68.2874.4477.1980.2682.51
ReFract72.2077.5281.8483.9785.55
ViT-B/16 → ViT-L/14Logit residual prediction75.8679.1683.5485.6986.93
ReFract80.5583.7286.4187.5688.72

ReFract consistently outperforms this baseline (+3.0 to +4.7 points on ViT-B/16 → ViT-B/16 and +1.8 to +4.7 on ViT-B/16 → ViT-L/14). Together with ReFract-Joint, this shows that learning block-specific targets from the decomposed donor residual offers a meaningful advantage over directly regressing the logit residual.

Target-only few-shot adaptation

More importantly, the paper lacks comparisons with simpler alternatives, such as directly predicting the teacher–target logit residual from frozen target activations or performing target-only few-shot adaptation. These baselines are important for establishing whether the proposed mechanism offers meaningful advantages over standard supervised residual prediction.

Also, in the Questions section:

Could the authors include additional baselines, particularly direct teacher–target logit residual prediction and target-only few-shot adaptation under the same calibration budget, to better establish the advantages of REFRACT?

We thank the reviewer for this suggestion, which was also raised by Rev. 8wHE and Rev. kGVk. Following it, we ran an experiment in which all methods share the same data budget, i.e., the same number of examples per class for each task. We evaluate full fine-tuning, LoRA fine-tuning and knowledge distillation (with full fine-tuning of the target model). We refer the reviewer to our response to Rev. 8wHE ("Comparison with direct receiver adaptation") for the complete discussion, and report the results below for convenience.

SettingMethodK=1K=2K=5K=10K=20
ViT-B/16 → ViT-B/16Full FT74.6181.5384.9587.3388.60
LoRA74.9679.4684.0686.5888.87
KD (Full FT)73.8079.1184.3487.5389.73
ReFract72.2077.5281.8483.9785.55
ViT-B/16 → ViT-L/14Full FT77.9882.8087.1389.7891.00
LoRA78.1380.4286.0189.0391.33
KD (Full FT)79.6882.6187.7090.0691.87
ReFract80.5583.7286.4187.5688.72

In short, methods that adapt the target backbone through multi-step gradient descent generally achieve higher accuracy as the data budget grows: in the same-architecture setting, they outperform ReFract at every budget (by 1.6 to 4.2 points; the gap narrows to at most 2.2 points when excluding SVHN, as discussed in our response to Rev. 8wHE), while in the cross-architecture setting full fine-tuning and KD surpass ReFract only from K=5 onward, and LoRA only from K=10. In the few-shot regime, instead, ReFract surpasses full fine-tuning, LoRA, and KD at K=1 and K=2 in the cross-architecture setting (80.55 vs. 77.98, 78.13, and 79.68 at K=1; 83.72 vs. 82.80, 80.42, and 82.61 at K=2), through closed-form fitting and without backpropagation through the target backbone.

We thank you again for your constructive feedback. As the discussion period is coming to an end, we hope that our clarifications on the methodological novelty, controlled ablations isolating the benefits of decomposition, and additional comparisons with simpler baselines have addressed your main concerns. Please let us know if anything remains unclear. We sincerely appreciate your time and consideration.

Reviewer_kGVkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes REFRACT, a cross-model task transfer framework based on decomposable fine-tuning residuals. By leveraging NTK-based linearized fine-tuning, the method decomposes functional residuals into additive block-wise contributions, which are transported and reconstructed using lightweight predictors on a frozen target model. Experiments across vision classification, natural language inference, and mathematical reasoning demonstrate competitive performance over existing model re-basin approaches.

Critical strengths

S1. Meaningful problem and innovative approach. Cross-model task transfer is an important research problem. The proposed shift from parameter-space alignment to NTK-based functional residual decomposition offers an interesting and theoretically motivated perspective.

S2. Simple and effective framework with strong empirical results. REFRACT requires no target backbone updates or source-model inference and achieves substantial improvements over existing re-basin methods across several vision and language transfer settings.

S3. Comprehensive experimental analysis. The paper evaluates different architectures, calibration budgets, and design choices, including decomposition granularity, predictor variants, and block smoothing, providing useful empirical support for the proposed framework.

Critical weaknesses

W1. Insufficient isolation of the core decomposition mechanism. The current comparisons do not fully disentangle the benefits of exact NTK-based decomposition from those of multi-layer features and multiple linear predictors, leaving the source of the improvements unclear.

W2. Limited evidence of practical advantages over direct target adaptation. REFRACT still requires labeled calibration data, but lacks sufficient comparisons with LoRA, lightweight adapters, or knowledge distillation under matched budgets, as well as an end-to-end computational cost analysis.

W3. Limited generality of the framework. Exact block-wise decomposition requires linearized source fine-tuning, restricting its applicability to existing conventionally fine-tuned models. Moreover, evaluations primarily focus on classification, with limited evidence for complex generative capabilities.

Questions for authors

Q1. Could the authors further isolate the contribution of exact NTK-based decomposition through randomized residual partitions, shuffled block correspondences, and controlled comparisons using identical target features and predictor capacities?

Q2. How does REFRACT compare with direct target adaptation using LoRA, lightweight adapters, or knowledge distillation under matched data and computational budgets?

Q3. Can the framework be extended to conventionally fine-tuned checkpoints and maintain its advantages on more diverse generative tasks and cross-family model transfers?

4 comments

We thank Reviewer kGVk for their careful assessment and constructive feedback.

Isolating the core decomposition mechanism

The current comparisons do not fully disentangle the benefits of exact NTK-based decomposition from those of multi-layer features and multiple linear predictors, leaving the source of the improvements unclear.

We thank the reviewer for raising this concern. To address it, we evaluated ReFract-Joint, an ablation that concatenates the same multi-layer receiver features and fits a single linear predictor to the global residual. This variant matches ReFract in input features and predictor parameter count, but replaces block-specific residual supervision with a single global regression objective.

SettingMethodK=1K=2K=5K=10K=20
ViT-B/16 → ViT-B/16ReFract-Joint71.1073.5077.2079.4081.00
ReFract72.2077.5281.8483.9785.55
Δ (percentage points)+1.10+4.02+4.64+4.57+4.55
ViT-B/16 → ViT-L/14ReFract-Joint78.0980.1082.4683.9485.27
ReFract80.5583.7286.4187.5688.72
Δ (percentage points)+2.46+3.62+3.95+3.62+3.45

ReFract outperforms ReFract-Joint in both settings and at every budget, with gains of 1.10–4.64 percentage points. These results indicate that access to richer receiver features and greater predictor capacity alone does not explain ReFract’s advantage, and support the use of structured, block-specific residual supervision. We have added this comparison to the main paper (Sec. 5, "Joint vs. decomposed prediction", Fig. 4b, covering both vision settings), with complete results in App. C.6.


Also, in the Questions section:

Q1. Could the authors further isolate the contribution of exact NTK-based decomposition through randomized residual partitions, shuffled block correspondences, and controlled comparisons using identical target features and predictor capacities?

We thank the reviewer for the suggestion. Controlled comparisons using identical target features and predictor capacities are covered by the ReFract-Joint ablation above.

Furthermore, inspired by the first two points raised by the reviewer -- i.e., randomized residual partitions and shuffled block correspondences -- we investigate whether the assignment of decomposed residuals to target blocks affects transfer performance. To this end, we ran an experiment in which the per-block residuals are assigned to the target network in a different order, rather than following the depth-wise correspondence adopted in our paper. Specifically, we evaluate: (1) reversed order, where the residual of the last source block is assigned to the first target block, and so on; and (2) random permutation, with results averaged over five permutations. All variants, including the ReFract reference, use independent fitting (ρ=0\rho = 0) to remove the confounding effect of error compensation in subsequent layers.

SettingMethodK=1K=2K=5K=10K=20
ViT-B/16 → ViT-B/16Reversed blocks69.6272.5776.4378.2479.76
Shuffled blocks70.3073.9978.5380.6982.28
ReFract71.0974.9579.5381.6883.40
SettingMethodK=200K=300K=500
T5-base → T5-largeReversed blocks71.6373.1674.54
Shuffled blocks75.7077.0378.34
ReFract77.1578.0079.59

As shown, assigning residuals in depth order consistently yields the best performance, supporting relative depth as a useful inductive bias. However, random permutations remain competitive, trailing the ordered assignment by only 0.79–1.12 percentage points in vision and 0.97–1.45 points in language. This is encouraging: ReFract benefits from a meaningful block correspondence, but does not appear to require a precise alignment to achieve effective transfer. Such robustness could be particularly useful when source and target backbones organize task-relevant information differently across layers. Finally, reversing the order causes a larger degradation, indicating that systematic mismatches can still hinder transfer.

Comparison with direct target adaptation and computational cost

Q2. How does REFRACT compare with direct target adaptation using LoRA, lightweight adapters, or knowledge distillation under matched data and computational budgets?

We thank the reviewer for this suggestion. We compared full fine-tuning, LoRA, and knowledge distillation (with full target fine-tuning) under a matched data budget, and refer the reviewer to our response to Rev. 8wHE ("Comparison with direct receiver adaptation") for the complete results. In brief, direct adaptation generally achieves higher accuracy as the data budget grows, while in the few-shot regime ReFract surpasses full fine-tuning, LoRA, and KD at K=1 and K=2 in the cross-architecture setting, through closed-form fitting and without backpropagation through the target backbone.

Generality of the framework

Exact block-wise decomposition requires linearized source fine-tuning, restricting its applicability to existing conventionally fine-tuned models. Moreover, evaluations primarily focus on classification, with limited evidence for complex generative capabilities.

Also, in the Questions section:

Q3. Can the framework be extended to conventionally fine-tuned checkpoints and maintain its advantages on more diverse generative tasks and cross-family model transfers?

Conventionally fine-tuned donors

We thank the reviewer for this point. Our decomposition framework originates from donors fine-tuned in tangent space. However, at the cost of an approximation, it can be readily extended to conventional non-linear fine-tuning. Indeed, the block-wise decomposition can be extended to conventionally fine-tuned donors through the fundamental theorem of calculus: DA(x)=∑ℓ∫01Jℓ(θA0+t ΔθA) ΔθA,ℓ dtD_A(x) = \sum_\ell \int_0^1 J_\ell(\theta_A^0 + t\,\Delta\theta_A)\,\Delta\theta_{A,\ell}\,dt, where JℓJ_\ell is the Jacobian with respect to block ℓ\ell and ΔθA,ℓ\Delta\theta_{A,\ell} its parameter update.

We approximate each block’s contribution by replacing the integral with an average over nn Monte Carlo samples: D^A,ℓ(x)=1n∑i=1nJℓ(θA0+ti ΔθA) ΔθA,ℓ\hat D_{A,\ell}(x) = \frac{1}{n}\sum_{i=1}^n J_\ell(\theta_A^0 + t_i\,\Delta\theta_A)\,\Delta\theta_{A,\ell}, with ti∼U[0,1]t_i \sim \mathcal{U}[0,1]. For non-linear donors, only residual extraction changes; the remaining ReFract pipeline is unchanged. In the linearized regime, the Jacobian is constant and a single evaluation is exact.

As a preliminary experiment, we tested this variant on ViT-B/16 → ViT-L/14 with n=4n = 4. We compare it with a single global ridge regressor (ReFract-Ridge) and an MLP (ReFract-MLP), both fitted using the same non-linearly fine-tuned donor without residual decomposition.

Donor FTPredictorK=1K=2K=5K=10K=20
Non-linearGlobal ridge (ReFract-Ridge)75.9977.2779.5581.1282.37
Non-linearMLP (ReFract-MLP)78.1080.3182.6283.6184.57
Non-linearBlock-wise, path-integrated (new)77.7080.3982.9084.3585.60

The block-wise variant outperforms global ridge regression by 1.7–3.4 percentage points and matches or improves upon the MLP from K=2 onward, with a gain of 1.0 point at K=20. These results support the applicability of decomposed supervision to conventionally fine-tuned models. They also suggest that decomposition makes the prediction problem more amenable to simple linear models: block-wise regressors fitted in closed form achieve performance comparable to or better than a non-linear predictor of the global residual.

We also assess the Monte Carlo approximation on ViT-B/16 by comparing the reconstructed residual D^A(x)=∑ℓD^A,ℓ(x)\hat D_A(x) = \sum_\ell \hat D_{A,\ell}(x) with the true non-linear residual DA(x)=f(x;θA0+ΔθA)−f(x;θA0)D_A(x) = f(x; \theta_A^0 + \Delta\theta_A) - f(x; \theta_A^0) on 100 randomly sampled test examples per dataset. The relative squared error ∥D^A−DA∥2/∥DA∥2\|\hat D_A - D_A\|^2 / \|D_A\|^2 is 0.79% on average across the eight datasets (at most 2.51%, on SVHN), and adding either residual to the pretrained output yields the same accuracy on every dataset.

Generative tasks

To complement the GSM8K mathematical reasoning results (Sec. 4 and App. C.10), we evaluated instruction-following transfer on IFEval. We fine-tuned the pretrained meta-llama/Llama-3.2-3B donor with LoRA in the linearized NTK regime, then used ReFract to transfer the adaptation to the pretrained meta-llama/Llama-3.2-1B receiver while keeping its backbone frozen.

IFEval reports the share of prompts with all instructions followed (prompt-level) and of individual instructions followed (instruction-level), under strict and loose scoring.

Model or comparisonPrompt strictPrompt looseInstruction strictInstruction loose
Pretrained 1B receiver (BB)9.411.813.916.0
Fine-tuned 3B donor (AftA_{\mathrm{ft}})24.526.436.939.0
Rebased 1B receiver (BrebasedB_{\mathrm{rebased}})19.520.233.134.6
Donor score recovered (%)79.676.589.788.7

ReFract improves the pretrained receiver's strict and loose prompt-level accuracy by 10.1 and 8.4 percentage points, and its strict and loose instruction-level accuracy by 19.2 and 18.6 points, respectively. At the instruction level, the rebased 1B receiver recovers nearly 90% of the fine-tuned 3B donor's score under both scoring protocols. Together with GSM8K, these results show that ReFract can transfer adaptations for both mathematical reasoning and instruction following generative tasks.

We thank you again for your constructive feedback. As the discussion period is coming to an end, we hope that our additional ablations on block-wise decomposition, comparisons with direct adaptation, and experiments on non-linear donors and generative tasks have addressed your main concerns. Please let us know if anything remains unclear. We sincerely appreciate your time and consideration.

Dear Reviewer,

Please take a moment to review the authors' rebuttal and provide your feedback on whether your concerns have been adequately addressed. Thank you for your time and effort.

Reviewer_8wHEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ReFract, a method for transferring task-specific capabilities between models. It decomposes a linearized source model’s fine-tuning-induced representation change into additive block contributions, transports these contributions, and fits linear predictors using a frozen target model’s intermediate features. The correction aims to match the fine-tuned source’s predictions, while deployment requires only the target model and fitted predictors. Experiments cover vision classification, natural language inference, and mathematical reasoning.

Critical strengths
  • The method supports transfer across different architectures without matching model parameters. Closed-form predictor fitting avoids updating the target backbone, and the source model is unnecessary at inference.

  • Experiments cover vision classification, natural language inference, and mathematical reasoning. The paper reports improvements over evaluated transfer baselines and includes ablations of source linearization, decomposition granularity, and residual smoothing.

Critical weaknesses
  • Although the paper claims to transfer functional residual, its objective is to match the source model’s predictions, which is more like a distillation objective, instead of only transferring the fine-tuned change TADA(x)T_AD_A(x). This might remove the receiver's pretrained capability zB0(x)z_B^0(x). Also, if we just want to match the source model’s predictions, why not use the source model directly or rely on distillation?

  • The ablations do not fully isolate the benefit of the proposed decomposition. Global ridge and MLP baselines use final-layer features, whereas ReFract accesses intermediate features and, for text, additional segment pooling. Consequently, the gains may partly reflect richer receiver features rather than the donor’s decomposed residuals. A direct distillation baseline using identical features, capacity, and regularization is missing.

  • The practical advantage over directly adapting the receiver is insufficiently established. The main method requires a donor trained in tangent space and repeated Jacobian-vector products to construct its targets. The efficiency evaluation primarily measures vision inference, leaving the preparation and fitting costs unclear. Comparisons with receiver fine-tuning or parameter-efficient adaptation under the same calibration budget would strengthen the practical claims. The paper also assumes linearized fine-tuning and fixed linear heads, which would also limit the fine-tuning performance.

Questions for authors
  • Why do we want to match the source model’s predictions instead of transferring the fine-tuned change TADA(x)T_AD_A(x)?

  • What are the donor preparation, residual extraction, and receiver fitting costs? Under the same data budget, when does this workflow outperform direct receiver adaptation in accuracy and total computation?

5 comments

We thank Reviewer 8wHE for their careful assessment and constructive feedback.

Distillation vs. residual transfer

Although the paper claims to transfer functional residual, its objective is to match the source model's predictions, which is more like a distillation objective, instead of only transferring the fine-tuned change TADA(x)T_A D_A(x). This might remove the receiver's pretrained capability zB0(x)z^0_B(x).

We thank the reviewer for highlighting this point.

Our goal is to transfer the functional residual DA(x)D_A(x) induced by fine-tuning the source model AA and use it to estimate a residual DB(x)D_B(x) that would result from fine-tuning the target model BB on the same task. To estimate DB(x)D_B(x), we formulate the transfer as an optimization problem that encourages the transported residual to reproduce the source model’s fine-tuning correction TADA(x)T_A D_A(x) in logit space.

We acknowledge that, in this respect, the objective is the same as that of knowledge distillation: one could indeed rely on distillation to match these predictions. The key difference lies in how this objective is attained. ReFract transfers the knowledge without retraining the target model: it requires neither a full gradient-descent optimization of the target backbone nor access to the entire training set, but only a few calibration samples. In this sense, ReFract can be read as a backpropagation-free form of distillation, which reuses the knowledge already encoded in the source residual instead of re-learning it from data.

Please note that transferring fine-tuning residuals through lightweight operations without retraining is central to research on model re-basin for fine-tuned models [a, b, c, d], as discussed in the Introduction of our paper.

As for using the source model directly, we address this point in our answer to the related question below.

Also, if we just want to match the source model's predictions, why not use the source model directly or rely on distillation?

To address this concern, imagine that a new backbone is released in November 2026, offering stronger general performance, better out-of-distribution robustness, improved safety, or reduced bias. Suppose we already have a source model fine-tuned in January 2026 on a specific downstream task. The goal of model re-basin, and ReFract in particular, is to enable an upgrade to the new backbone with two objectives:

  1. Quickly transfer the specialist knowledge without gradient descent, preserving downstream task performance.
  2. Retain the advantages of the new backbone, such as better out-of-distribution performance.

Deploying the fine-tuned source directly would forgo these advantages. The second objective also directly relates to the reviewer’s concern about preserving the receiver’s pretrained capabilities zB0(x)z_B^0(x). We acknowledge that this motivation was not sufficiently emphasized in the main paper.

To support this point, we measure the accuracy both on the 8 target tasks (average) and on the ImageNet validation set, which we use as a proxy for general capabilities (ViT-B/16 → ViT-L/14, K = 1 and K = 20).

Model8-Vision AVGImageNet val
θA\theta_A zero-shot (ViT-B/16)66.9977.10
θA\theta_A fine-tuned, linearized (ViT-B/16)92.3472.79
θB\theta_B zero-shot (ViT-L/14)73.8882.70
θB\theta_B + ReFract (ViT-L/14) K = 180.5581.35
θB\theta_B + ReFract (ViT-L/14) K = 2088.7281.52

For the fine-tuned source and the ReFract-edited models, ImageNet accuracy is averaged over the 8 task-specific models.

Full REFRACT: top-1 accuracy (%) on the ImageNet validation set for each steering dataset, using K = 20 examples per class.

Steering datasetCarsRESISC45EuroSATSUN397SVHNDTDMNISTGTSRBAverage
ImageNet accuracy (K=20)78.2082.1382.2079.3382.7382.7382.7382.1081.52

As shown, the ViT-L/14 edited with ReFract outperforms the fine-tuned ViT-B/16 on the ImageNet validation set by almost 9 points (81.35–81.52 vs. 72.79), and retains most of its zero-shot ImageNet accuracy (−1.2 to −1.4 points w.r.t. 82.70), while improving on the target tasks from 73.88 to 80.55 (K = 1) and 88.72 (K = 20). By contrast, fine-tuning the source model costs 4.3 points on ImageNet w.r.t. its zero-shot counterpart (77.10 → 72.79). The per-dataset breakdown confirms that this behavior is stable across steering datasets and budgets.

This indicates that ReFract transfers the task-specific correction while largely preserving the general capabilities of the stronger receiver backbone, which would be lost by deploying the source model directly. We have added this analysis to the main paper (Sec. 5, "Retention of general capabilities", Fig. 4c), where it is presented as a trade-off between in-distribution (8-Vision) and out-of-distribution (ImageNet) accuracy.

Isolating the benefit of the decomposition

The ablations do not fully isolate the benefit of the proposed decomposition. Global ridge and MLP baselines use final-layer features, whereas ReFract accesses intermediate features and, for text, additional segment pooling. Consequently, the gains may partly reflect richer receiver features rather than the donor's decomposed residuals.

To isolate the role of the decomposition from that of richer receiver features, we ran a controlled ablation with identical features and capacity. In particular, starting from our method, we concatenate the multi-layer features used by ReFract, [HB,1(x);… ;HB,L(x)][H_{B,1}(x); \dots; H_{B,L}(x)], and fit a single ridge regressor that predicts the global (non-decomposed) residual, transported with the same Stage 1. Since ∑ℓWℓHB,ℓ=[W1,…,WL] [HB,1;… ;HB,L]\sum_\ell W_\ell H_{B,\ell} = [W_1, \dots, W_L]\,[H_{B,1}; \dots; H_{B,L}], this variant has the same features and the same number of parameters as ReFract, and differs only in whether the target residual is decomposed across blocks. We refer to this variant as ReFract-Joint.

SettingMethodK=1K=2K=5K=10K=20
ViT-B/16 → ViT-B/16ReFract-Joint71.1073.5077.2079.4081.00
ReFract72.2077.5281.8483.9785.55
ViT-B/16 → ViT-L/14ReFract-Joint78.0980.1082.4683.9485.27
ReFract80.5583.7286.4187.5688.72

Even with identical features and capacity, ReFract outperforms ReFract-Joint by +1.1 to +4.6 points on ViT-B/16 → ViT-B/16 and by +2.5 to +4.0 points on ViT-B/16 → ViT-L/14. Together, these results indicate that the advantage of our method does not stem solely from the availability of richer receiver features, but also from learning block-specific targets derived from the donor's decomposed residual. We have added this comparison to the main paper (Sec. 5, "Joint vs. decomposed prediction", Fig. 4b, covering both vision settings), with complete results in App. C.6.

A direct distillation baseline using identical features, capacity, and regularization is missing.

To further address the reviewer's concern, we introduce a direct distillation baseline. Given the features HB(x)H_B(x) extracted from the frozen target model, we fit the head of the target model so that it directly matches the predictions of the fine-tuned source model:

min⁡Δ ∣∣(zAft(x)−zB0(x))−ΔHB(x)∣∣2+λ∣∣Δ∣∣F2.\min_{\Delta}\ || \big(z_A^{ft}(x) - z_B^0(x)\big) - \Delta H_B(x) ||^2 + \lambda || \Delta ||^2_F .

This distillation objective can be solved in closed form and, like our method, requires neither backpropagation through the target backbone nor the source model at inference time. The key difference is that it distills the source predictions as a whole, without exploiting the decomposition of the donor's residual.

SettingMethodK=1K=2K=5K=10K=20
ViT-B/16 → ViT-B/16Distillation baseline68.2874.4477.1980.2682.51
ReFract72.2077.5281.8483.9785.55
ViT-B/16 → ViT-L/14Distillation baseline75.8679.1683.5485.6986.93
ReFract80.5583.7286.4187.5688.72

As shown, ReFract consistently outperforms the distillation baseline across settings and budgets (+3.0 to +4.7 points on ViT-B/16 → ViT-B/16, and +1.8 to +4.7 points on ViT-B/16 → ViT-L/14).

Comparison with direct receiver adaptation

The practical advantage over directly adapting the receiver is insufficiently established. [...] Comparisons with receiver fine-tuning or parameter-efficient adaptation under the same calibration budget would strengthen the practical claims.

Also, in the Questions section:

Under the same data budget, when does this workflow outperform direct receiver adaptation in accuracy and total computation?

Following the reviewer's suggestion, we ran an experiment in which all methods share the same data budget, i.e., the same number of examples per class for each task. We evaluate full fine-tuning, LoRA fine-tuning and knowledge distillation (with full fine-tuning of the target model).

SettingMethodK=1K=2K=5K=10K=20
ViT-B/16 → ViT-B/16Full FT74.6181.5384.9587.3388.60
LoRA74.9679.4684.0686.5888.87
KD (Full FT)73.8079.1184.3487.5389.73
ReFract72.2077.5281.8483.9785.55
ViT-B/16 → ViT-L/14Full FT77.9882.8087.1389.7891.00
LoRA78.1380.4286.0189.0391.33
KD (Full FT)79.6882.6187.7090.0691.87
ReFract80.5583.7286.4187.5688.72

As shown, methods that adapt the target backbone through multi-step gradient descent generally achieve higher accuracy as the data budget grows: in the same-architecture setting, full fine-tuning, LoRA, and KD outperform ReFract at every budget (by 1.6 to 4.2 points), while in the cross-architecture setting full fine-tuning and KD surpass ReFract only from K=5 onward, and LoRA only from K=10. This is unsurprising: when sufficient data and computation are available, directly updating the receiver via multi-step gradient descent offers greater flexibility to optimize downstream performance. We therefore view direct adaptation as a useful performance reference for model re-basin approaches like ours.

Additional analysis: excluding SVHN. ReFract struggles where the frozen target features are poorly informative, most notably on SVHN and, to a lesser extent, GTSRB (14.7–19.2 and 10.1–13.3 points below full fine-tuning): predicting the residual from frozen features cannot recover information they do not encode, whereas updating the backbone can. Since SVHN weighs considerably on the average, we also report the average over the remaining seven tasks (ViT-B/16 → ViT-B/16):

MethodK=1K=2K=5K=10K=20
Full FT74.7881.7785.3287.2388.70
LoRA75.4779.7084.3786.9288.94
KD (Full FT)75.9979.9284.5187.5589.56
ReFract74.1479.9384.3486.0587.32

Excluding SVHN, ReFract trails the methods that fine-tune the backbone by at most 2.2 points and is on par with LoRA and KD at K=2 and K=5 (−0.17 to +0.23 points), while requiring no gradient descent on the target model.

In the few-shot setting, ReFract not only remains competitive but also surpasses full fine-tuning, LoRA, and KD at K=1 and K=2 in the cross-architecture setting (80.55 vs. 77.98, 78.13, and 79.68 at K=1; 83.72 vs. 82.80, 80.42, and 82.61 at K=2). These results suggest that the transferred residuals provide an informative, task-specific inductive bias that helps compensate for scarce calibration data, without requiring backpropagation through the target backbone.

Preparation and fitting costs

Starting from an available tangent-space ViT-B donor checkpoint, we measured the wall-clock time to produce a ViT-L/14 receiver ready for inference. We used DTD with K=10K=10 calibration images per class (470 images total). The cold run includes setup, residual extraction, receiver fitting, and saving.

Table 1. REFRACT preparation and fitting on DTD. We report mean ± SD where available over seeds 33, 42, and 54. We used one NVIDIA L40S GPU in FP32, an Intel Xeon E5-2680 v4 CPU, and an extraction batch size of up to 256. Ridge fitting ran on the CPU in float64 with eight threads.

StageTime or peak GPU memory
Residual extraction (including 13 group-wise JVPs)396.0 ± 5.5 s
Peak extraction GPU memory (allocated / reserved)7.93 / 13.87 GiB
Cold receiver fitting (Stages 1 and 2)0.37 s
Setup, cache writing, and saving25.8 s
Total cold time422.2 ± 14.5 s
Cached refit (loading and fitting only)2.1 s
Total cached refit time27.1 ± 9.6 s

Residual extraction accounts for 94% of the cold runtime. With cached residuals, loading and fitting take 2.1 s, and the complete refit takes about 27 s.

Table 2. Adaptation cost and accuracy on DTD at K=10K=10. Knowledge distillation (KD) uses SGD with batch size 16 for 50 epochs. Ready time excludes evaluation. Multipliers use the mean KD ready time as the reference.

MethodReady timePeak GPU memory (GiB, allocated / reserved)Storage per taskAccuracy (%)
Zero-shot26.8 ± 8.1 s1.63 / 1.67—66.54
KD (full receiver, 304M parameters)560.7 ± 31.7 s (1.00×)11.00 / 11.671.13 GiB80.24 ± 1.15
REFRACT, cold422.2 ± 14.5 s (0.75×)7.93 / 13.870.145 GiB79.37 ± 0.27
REFRACT, cached refit27.1 ± 9.6 s (0.05×)2.86 / 2.990.145 GiB79.37 ± 0.27

On DTD, cold REFRACT achieves 79.37% mean accuracy, 0.87 percentage points below full-receiver KD, 98.9% of KD’s accuracy, while using 75% of its adaptation time and about one eighth of its per-task storage. With residuals already cached, REFRACT achieves the same accuracy in about 5% of the KD ready time.

Table 3. Receiver inference latency on DTD. Batch-1 times are per forward pass. Batch-256 times are per image.

MethodBatch 1 (ms)Batch 256 (ms/image)
Receiver forward (reference for zero-shot and KD)10.16 ± 0.186.16 ± 0.08
REFRACT11.08 ± 0.096.69 ± 0.01

The REFRACT correction adds 9.1% to mean inference latency at batch size 1 and 8.6% at batch size 256.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.