- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes ARIADNE, a method for transferring the effects of fine-tuning between Transformers with different depths and widths. Using paired calibration inputs, the method collects intermediate activations from the source base model, the source fine-tuned model, and the target base model. It establishes module correspondences based on relative depth, uses local Procrustes mappings to transform the activation displacements induced by source fine-tuning into the target coordinate system, and then fits parameter updates for the final projection layers of the target MLPs via closed-form ridge regression. The resulting updates can be written directly into the target model, without activation interventions at inference time. The default activation-fitting stage requires neither labels nor backpropagation, whereas the gradient variant requires the corresponding supervision.
Critical strengths
- Transferring fine-tuning displacements is well motivated: The method preserves the target base representations and transforms the changes induced by source fine-tuning into target parameter updates, avoiding the direct transfer of source parameters with incompatible shapes. The relevant experimental results show that fitting the absolute fine-tuned endpoint performs worse than fitting the fine-tuning displacement, providing empirical support for the choice of transfer target.
- The vision experiments demonstrate gains in both accuracy and fitting efficiency: In both vision transfer directions, the default method achieves substantially higher average accuracy than the parameter-transfer baselines included in the comparison. The appendix also reports results under different budgets and across three calibration runs, providing substantial positive evidence from the vision experiments.
Critical weaknesses
-
Is the closed-form solution for dimensionality-reducing alignment consistent with the stated objective? When the source width exceeds the target width, Equation (1) on page 4 requires the mapping to have orthonormal columns. Equation (2) then directly uses the product of the SVD factors of the source–target cross-matrix as the mapping. This solution maximizes the inner product between the mapped source activations and the target activations. However, the squared-error objective also includes the squared norm of the mapped source activations, which generally varies with the mapping under the orthonormal-column constraint in the dimensionality-reducing case. The authors therefore appear not to have explained the conditions required for Equation (2) to be an optimal solution to Equation (1), or whether these conditions are satisfied in the actual method. Could the authors clarify these points?
-
Has the accuracy of the fixed-input approximation been validated after assembling the updates? The paper restricts the expression for the change in component outputs (Equation (9)) to a fixed input bank. It further explains that each layer is fitted independently using inputs from the target base model, and that the updates are assembled only after all fitting is complete. However, once upstream layers are updated, the inputs actually received by downstream components will generally differ from those used during fitting. Have the authors considered this effect? Could they report, after all updates have been assembled and using the same evaluation inputs, the error between the actual output displacement of each target module and its corresponding desired displacement, and compare it with the fitting error under fixed inputs? This would quantify the effect of input changes on local displacement matching.
Questions for authors
-
When the source width exceeds the target width, under what conditions does the SVD solution in Equation (2) minimize Equation (1), and are these conditions satisfied in the actual implementation?
-
After all updates have been assembled, how much does the error between the actual and desired module output displacements change compared with the fitting error under fixed inputs?
2 comments
W2/Q2. Fixed-input approximation after assembly.
The reviewer is right that each regression in Eq. (9) is solved on a fixed input bank collected from the target base model. Once all updates are assembled, block instead receives inputs already changed by the updates of blocks . Its realized output displacement can therefore differ from the fixed-input fit. To quantify this gap, as the reviewer suggests, we compare the default independent fitting with a sequential variant that accounts for upstream updates:
Independent fitting (default). Each block is fitted on inputs from the target base to reproduce the full displacement , as if upstream blocks were unchanged (Eqs. 8–9).
Sequential fitting. Blocks are fitted one at a time, from the first to the last. Each block is fitted on the inputs it actually receives once the upstream ARIADNE updates are applied, and reproduces only what is still missing: , where is the displacement after block due to the upstream updates alone.
We measure two errors for each target block , both normalized by , and evaluated on held-out images disjoint from the calibration set:
- Local error: how well the block’s own update reproduces its target on the inputs it was fitted on, i.e., for independent fitting and for sequential fitting.
- In-network error: , where is the displacement actually realized after block , relative to the target base, when all updates are applied together.
For sequential fitting, the realized displacement is , so local and in-network errors coincide by construction; this does not imply zero error with respect to .
| Fitting | Local error | In-network error | |
|---|---|---|---|
| B/16→L/14, block 1 | both | 0.29 | 0.29 |
| B/16→L/14, median over blocks | independent | 0.76 | 2.48 |
| sequential | 0.99 | 0.99 | |
| L/14→B/16, block 1 | both | 0.23 | 0.23 |
| L/14→B/16, median over blocks | independent | 0.82 | 1.63 |
| sequential | 0.93 | 0.93 |
Independent updates accumulate. At the first block, no upstream update exists and the two errors coincide. Deeper in the network, independently fitted updates add up: each block reproduces the full displacement on top of what upstream blocks have already produced. With independent fitting, the error grows from 0.76 (local) to 2.48 (in-network) in extension and from 0.82 to 1.63 in reduction.
Why sequential fitting has a higher local error. Sequential fitting removes this accumulation (0.99 and 0.93), but its local error is higher than in the independent case. Independent fitting never has to correct errors coming from upstream blocks: each block only matches on the base inputs. Sequential fitting must instead also absorb (i) the residual error of the upstream updates and (ii) the effect of those updates on the non-linear components (normalization, attention, and MLP activations) of the subsequent blocks, a correction that grows deeper in the network. Since an update of the MLP output projection has only partial expressive power, as Adila et al. [1] also show for MLP-output interventions compared with interventions on the full residual stream, each block can satisfy this growing request only partially, which raises its local error.
However, better displacement matching does not uniformly improve accuracy: sequential fitting increases reduction accuracy from 78.96% to 81.06%, but decreases extension accuracy from 84.54% to 82.60%. We therefore retain independent fitting as the default, since sequential fitting does not consistently improve performance and introduces dependencies between block fits.
We will incorporate these diagnostics and the sequential comparison into the revised paper.
[1] Adila et al. Weight Updates as Activation Shifts: A Principled Framework for Steering. ICML, 2026.
We thank the reviewer for the valuable feedback. Below, we address the two mathematical concerns: we correct the stated alignment objective without changing the implementation or reported results, and provide new held-out diagnostics and a sequential-fitting comparison to quantify the effect of assembling independently fitted updates. We will incorporate these clarifications and results into the revised paper.
W1/Q1 Solution to the reduction case.
The reviewer is correct: Eq. (2) does not generally minimize the squared-error objective in Eq. (1) when the source is wider than the target. This is an error in the mathematical description in the paper. Our implementation directly computes the polar factor of the centered source–target cross-covariance, as specified in Eq. (2), in both directions; all reported experiments use this construction.
More precisely, writing X=Sπ(j)base and Y=Tjbase, the implemented alignment solves Qj∈argmaxR∈Pds,dttr(R⊤X⊤Y). For the thin SVD (X⊤Y=UjΣjVj⊤), a solution is Qj=UjVj⊤.
∥XR−Y∥F2=tr(R⊤X⊤XR)+∥Y∥F2−2tr(R⊤X⊤Y). When ds≤dt, the constraint RR⊤=I makes the first term constant, so the two objectives are equivalent. When ds>dt, the constraint is R⊤R=I, and the first term generally varies with R. Isotropic source covariance would suffice for equivalence, but we do not assume it. In reduction, Eq. (2) therefore maximizes aggregate source–target cross-covariance under a semi-orthogonality constraint, but need not minimize squared error. The resulting map does not amplify the source displacement’s norm. We believe this is the right objective for transfer: we want to know where the fine-tuning change should go in the target, not to stretch it to fit the target activations. Consistent with this, maps fitted to minimize reconstruction error transfer worse in reduction: an unconstrained linear map (ridge, Table 2) reaches 76.52%, and a two-layer MLP (our response to Reviewer m37t) 76.33%, versus 79.21% for Qj in the same run.
We will correct Eq. (1) in the main paper to state the cross-covariance maximization objective and explicitly explain this distinction between expansion and reduction. Eq. (2), the implementation, and the reported numerical results remain unchanged.