acceptodds
Withdrawn from ICLR 2027

Towards Understanding Feature Learning in Parameter Transfer

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Parameter transfer is a central paradigm in transfer learning, enabling knowledge reuse across tasks and domains by sharing model parameters between upstream and downstream models. However, when only a subset of parameters from the upstream model is transferred to the downstream model, there remains a lack of theoretical understanding of the conditions under which such partial parameter reuse is beneficial and of the factors that govern its effectiveness. To address this gap, we analyze a setting in which both the upstream and downstream models are ReLU convolutional neural networks (CNNs). Within this theoretical framework, we characterize how the inherited parameters act as carriers of universal knowledge and identify key factors that amplify their beneficial impact on the target task. Furthermore, our analysis provides insight into why, in certain cases, transferring parameters can lead to lower test accuracy on the target task than training a new model from scratch. In addition, our results highlight the importance of selecting appropriate parameters for transfer, suggesting that inheriting weights activated by the signals is crucial for achieving positive transfer. Numerical experiments and real-world data experiments are conducted to empirically validate our theoretical findings.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Parameter Transfer, Transfer Learning, Learning Theory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 5 reviews, above 27% of the others.
Reviewer_xfoiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper analyzes the success/failure of transfer learning and proposes a theoretical framework for better understanding transfer dynamics and its success theoretically for two layer CNNs. It then presents experimental results with several datasets and deeper networks including both CNNs and ViTs to show experimentally the factors that impact the transfer, and show those are consistent with the theoretical findings.

Critical strengths

In my opinion, the theoretical framework, although limited to 2-layer CNNS, is the main strength of the paper, which contributes to better understanding the dynamic of transfer learning in this specific settings. It is also complemented by the several experiments the authors performed on various datasets and networks.

Critical weaknesses

While the theoretical analysis is useful, I have concerns about its practicality in real transfer learning settings and the novelty of some of the experiments. I agree with the authors that a theoretical understanding for transfer learning is lacking from the current literature, but I am not fully sure how the theoretical framework proposed in their setup can be of practical use for deeper modern networks from a practical point of view. Some experiments, for example that increasing upstream data size can help transfer (Figure 1.a), or that reducing upstream noise improves downstream performance (Figure 1.b), are already well-known in the literature. Similarly, it has been previously shown that transfer learning can have negative effects on unrelated tasks, for example in [1].

Does the theoretical framework applies to more complex settings, e.g. vision-language models or self-supervised learning ? Coud the authors shed some light on the practicality of their framework in these cases?

[1] Mensink, T., Uijlings, J., Kuznetsova, A., Gygli, M., & Ferrari, V. (2021). Factors of influence for transfer learning across diverse appearance domains and task types. IEEE Transactions on Pattern Analysis and Machine Intelligence

Questions for authors

See above.

Reviewer_YWdbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies a toy theoretical model for transfer learning on a two-layer CNN. The source and target tasks each have signals in a 2d subspace, with 1 direction shared across the task and 1 direction distinct in each task, and the noise in each task is a Gaussian vector restricted to the orthogonal complement of the 2d signal space of the task. The transfer is done by keeping top \alpha fraction of the learned weights. The main results are:

  • Theorem 4.2, which shows a phase transition of the generalisation performance on a new data point as a specific relationship between data size, signal size, noise size, data dimension and \alpha varies.

  • Theorem 4.3, which shows a similar transition for the case \alpha=0, i.e. no parameter transfer.

  • Proposition 4.4, which interprets the conditions for phase transitions in the two results above to understand when parameter transfer helps / hurts compared to no transfer.

  • Theorem 4.5, which says that under a theoretically worst-case transfer routine -- where, instead of keeping the top \alpha fraction of the learned weights, the kept weights are the part of the learned weights with zero shared signal component -- then parameter transfer never helps. The author uses this to conclude that any benefits from parameter transfer comes from weights that have learned the shared signal.

The author also claims (2nd paragraph, page 1) that "parameter transfer is intrinsically a question of network dynamics", "a rigorous theoretical characterization (of transfer) is missing", and they "provide the first feature-learning analysis of the training dynamics of partial parameter transfer", which seems to be the central novelty claim.

The authors then conduct numerical experiments on their toy model (Sec 6) and real data experiments (Sec 7) to confirm that increasing sample size of source task / reducing noise in source task / increasing shared signal strength all help with transfer (bottom of page 7, discussions in Sec 7, first paragraph of Sec 8).

Critical strengths
  • To the best of my knowledge, this is the first theoretical work that analyses transfer learning under a CNN setting
Critical weaknesses
  1. Possibly overclaimed theoretical contributions and novelty:

The paper's main claims are already covered for non-CNN models in a large body of theoretical works on transfer learning, which the paper seems not to addresses nor compare. In particular, there are many results that have already shown phase transitions and positive and negative transfers [1,3,4], and have already attributed transfer learning success to sample size, noise and size of shared signals [1,2,3,4].

While I think the CNN angle could be interesting, the paper barely discusses how the analysis of CNN is new or whether there is new insights to be gained compared to non-CNN models.

Instead, the novelty claim (and the title) anchors its contributions on feature learning and learning dynamics analysis. However, there is no discussion on why their model is in the feature learning regime (other than a unproved remark in line 233-234 about a condition 4.1, 4). Moreover, the results are fundamentally about generalisation error, and the use of "learning dynamics" seems to be just a standard optimisation based analysis (based on the proof sketch in Section 5) that is considered e.g. in [2]. Importantly, none of the feature learning / so-called "learning dynamics" analysis lead to further understanding on transfer learning that are already available via other tools and in the existing literature, e.g. the ones in [1,2,3,4] mentioned above.

  1. Mathematical assumptions are restrictive, under-explained and violated even in numerical experiments:

The theoretical model is very simple with strong assumptions. Although this is understandable as most of the setup is borrowed from an existing line of work on CNN analysis (e.g. [5]) which enables simplified analysis, a lot more explanations will help with the presentation.

My main issue, however, is with how the combination of their Condition 4.1 and the main results seem to undermine the utility of the result significantly. For instance, taking N1 N2 NN_1 ~ N_2 ~ N, σp,1 σp,2 σp\sigma_{p,1} ~ \sigma_{p,2} ~ \sigma_p, ∣∣u∣∣∼∣∣v2∣∣|| u || \sim || v_2|| for simplicity, the phase transition (in Theorem 4.2, but also see the quantity Γ\Gamma in Proposition 4.4) occurs when dd crosses (some constant) ×N∣∣u∣∣4σp4\times N \frac{|| u ||^4}{\sigma_p^4}, i.e. the interesting phase transition happens when

∣∣u∣∣4σp4∼dN\frac{|| u ||^4}{\sigma_p^4} \sim \frac{d}{N}

but Condition 4.1 requires d⪰N2d \succeq N^2 which means, for us to see the phase transition, we need the signal-to-noise ratio to at least satisfy ∣∣u∣∣4σp4⪰N\frac{|| u ||^4}{\sigma_p^4} \succeq N . This feels unreasonably restrictive. For instance, in the numerical experiments in Section 6, ∥u∥\| u \| is chosen to be in the single digits, σp\sigma_p is between single and double digits, whereas NN is on the order 100-1000.

[1] Phase Transitions in Transfer Learning for High-Dimensional Perceptrons, Oussama Dhifallah and Yue M. Lu, Entropy 2021

[2] An analytic theory of generalization dynamics and transfer learning in deep linear networks, Lampinen and Ganguli, ICLR 2019

[3] Features are fate: a theory of transfer learning in high-dimensional regression, Tahir, Ganguli and Rotskoff, ICML 2025

[4] Double Double Descent: On Generalization Errors in Transfer Learning between Linear Regression Tasks, Dar and Baraniuk, SIMODS 2022

[5] Benign overfitting in two-layer convolutional neural networks. Cao, Chen, Belkin and Gu. NeurIPS 2022

Questions for authors
  1. Can the authors provide a satisfatory comparison of their theoretical results and conclusions to the large body of theoretical transfer learning literature, and in particular what is new in their CNN analysis versus what people already know from existing theory?

  2. Can the author provide mathematically rigorous justifications for the line 233-234 that the model they considered is doing feature learning and not NTK, as well as how this distinction plays a role in the insights on transfer learning?

  3. Can the authors clarify and justify the mathematical assumptions used, which I understood to be very restrictive and also not respected by the numerical experiments (pointed out in 2) in "Critical Weaknesses"), but maybe I am missing something important?

Reviewer_6AiQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper analyzes gradient-descent training of a two-layer ReLU CNN on a signal/noise patch model with a shared signal and task-specific signals, where a fraction of upstream filters is inherited, and the rest are reinitialized. It derives a dimension-dependent phase transition in downstream test error relative to training from scratch, conditions for positive and negative transfer, and a claim that only signal-active filters yield gains. Synthetic and real-data experiments are provided.

Critical strengths
  1. The paper analyzes how inherited filters can carry both useful signal features and potentially harmful noise memorization into downstream training, using training dynamics rather than static generalization bounds. This proposes a potentially valuable explanation for positive and negative transfer beyond simply observing whether pretraining improves performance.

  2. The synthetic shared-signal experiments hold the downstream signal strength fixed while varying the shared component. This helps separate the effect of shared signal from task difficulty and makes those trends easier to interpret.

Critical weaknesses

W1: Several central theoretical claims require stronger justification such as:

  1. Condition 4.1 and κ_D (p. 32): The proof states that Condition 4.1 implies κ_D ≤ 0.1. But κ_D contains a term scaling with the ratio of the two tasks’ noise levels and the upstream sample size, which Condition 4.1 does not appear to control. Is an additional condition needed?
  2. Theorem F.12(3) and Eq. (F.20): The dimension threshold from the proof’s perturbation-norm calculation does not appear to follow from the theorem’s stated condition in all parameter regimes. Also, the perturbation is built from downstream noise only, yet the subsequent inner-product bound includes upstream noise coefficients at full strength. I could not reconcile this with the paper’s own cross-task coupling bound in Lemma F.2.
  3. Proposition 4.4(2): The negative-transfer conclusion needs a dimension range where the transfer and from-scratch guarantees hold simultaneously. It is unclear whether the stated assumptions ensure one, particularly for small inherited fractions.
  4. Theorem 4.5/F.13: The proof appears to show that the authors’ bound gives no improvement, not that actual test error is no better. The claimed necessity of signal-active inheritance therefore needs stronger justification.
  5. A smaller inconsistency: The Task 2 loss bound in Theorem F.12(1) appears to use upstream quantities (σₚ,₁, N₁). These are gaps in the presented arguments, not demonstrated counterexamples, but they lower my confidence in the central theoretical conclusions.

These are gaps in the presented arguments, not demonstrated counterexamples, but they lower my confidence in the central theoretical conclusions.

W2. The empirical evaluations do not sufficiently validate the proposed mechanisms, such as:

  1. The theoretical guarantees assume a dimension scaling at least quadratically with sample size (up to logarithmic factors). The synthetic experiments do not demonstrate that Condition 4.1 is satisfied. For example, some runs use d = 2000 with an upstream sample size of 1000. They show the predicted qualitative trends, but it is unclear whether they fall within the regime the guarantees cover.

  2. The real-data experiments show relevant transfer behavior on deeper architectures, but they do not isolate the mechanisms the theory emphasizes: shared vs. task-specific signal, inherited noise memorization, and signal-active filter selection. There is no controlled comparison of random, signal-active and signal-free inheritance.

  3. Some experiments, such as the text negative-transfer ones, transfer all parameters, unlike the partial-transfer setting analyzed. Clearer descriptions of what is transferred, plus seed counts and variance estimates, would strengthen reproducibility.

Questions for authors
  1. How does Condition 4.1 guarantee the required bound on κ_D despite its dependence on the cross-task noise ratio and upstream sample size?
  2. In Theorem F.12(3), how does the stated dimensionality threshold follow from the perturbation-norm calculation? Why does Equation (F.20) include upstream noise coefficients at full strength when the perturbation uses only downstream noise?
  3. Can the authors state the dimension range under which Proposition 4.4(2) holds, write out the derivation, and clarify whether additional assumptions (for example on the inherited fraction) are required?
  4. Can the claimed necessity of signal-active inheritance (Theorem F.13) be established for actual test error, not only the derived bound?
  5. Can the authors clarify how the synthetic experiments relate to the theoretical regime? Can they add a controlled comparison of random, signal-active and signal-free inheritance, with the signal-free set truly having no component along the shared signal?
  6. What exactly is transferred in the CIFAR (ResNet/VGG) experiments, and how many seeds and what variances were used for the real-data results?
Reviewer_CgSrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies partial parameter transfer in a two-layer ReLU CNN by decomposing each task into a shared signal, a task-specific signal, and noise. It analyzes how inherited filters carry useful shared information or memorized noise, derives generalization conditions for positive and negative transfer, and argues that signal-active filters are the most useful parameters to transfer

Critical strengths

The paper addresses an important and relatively underexplored theoretical question: how partial parameter transfer affects feature-learning dynamics rather than only final generalization bounds. The signal/noise coefficient framework is intuitive and technically substantial, and the paper connects the theory to synthetic and real-data experiments covering both positive and negative transfer.

Critical weaknesses

However, the main result relies on a highly restrictive additive decomposition u+vku + v_k, where a single vector uu is interpreted as universal knowledge. This substantially limits the external validity of the theory: transferable representations in modern neural networks are generally distributed across nonlinear, multidimensional, and layer-dependent feature spaces, and therefore cannot in general be reduced to one shared vector.

More importantly, the issue is not only that the model is idealized. Under the assumptions actually stated, the decomposition into universal and task-specific signals is not identifiable. The paper assumes only u⊥v1u \perp v_1 and u⊥v2u \perp v_2, while the main generalization criterion depends explicitly on ∥u∥4\| u \|^4. One can construct for which ua⊥vk,a,ua+vk,a=Le1,Pua+Pvk,a=P{spane1,e2}.u_a \perp v_{k,a}, \qquad u_a + v_{k,a} = L e_1, \quad P_{u_a} + P_{v_{k,a}} = P_\{span e_1, e_2 \}. Thus, changing aa leaves the complete source and target data distributions unchanged, but changes ∥ua∥4=L4a2\| u_a \|^4 = L^4 a^2. With the scaling used in the counterexample, Theorem 4.2 predicts asymptotically vanishing error for one value of aa and a constant error lower bound for another, although the two learning problems are distributionally identical. Therefore, the main theorem cannot hold under the stated assumptions.

The synthetic experiments avoid this issue by imposing mutual orthogonality of u,v1,v2u, v_1, v_2, suggesting that a stronger geometric assumption may have been intended. Under such an amended model, the proof strategy may be salvageable, but the theorem and its supporting decomposition must be rechecked. Even after such a repair, the conclusions should be explicitly restricted to an orthogonal latent-factor model rather than presented as a general account of transfer learning in neural networks. Overall, the technical analysis is promising, but the current formulation contains a major identifiability flaw and overstates the generality of the resulting theory.

Questions for authors
  1. The formal model assumes only u⊥v1u \perp v_1 and u⊥v2u \perp v_2, while the main theorem depends explicitly on || u ||. How is the decomposition into shared and task-specific signals uniquely identified from the data distribution?

  2. The synthetic experiments impose mutual orthogonality of u,v1,v2u, v_1, v_2, which is stronger than the theorem assumptions. Was v1⊥v2v_1 \perp v_2 intended as an additional assumption, and if so, which parts of the theorem and proof require it?

  3. Since the experiments mainly support qualitative trends, could the authors add a direct test of the predicted phase-transition scaling and compare random, signal-active, and signal-free parameter selection?

Reviewer_jHzNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This theory-driven paper examines the dynamics of feature learning under parameter transfer in a simplified setting. The author(s) considered a two-layer, convolutional-like neural network. In this network, the first layer has ReLU activation and shared weights, while the second layer is fixed. The theory for parameter transfer encompasses two datasets, each with (a) a shared and distinct part that is orthogonal to the shared part, and (b) the isotropic Gaussian noise directs to other orthogonal subspace. Based on concentration inequalities and in the over-parameterized regime with small initialization, this study derives bounds on the test error of the second dataset after training on the first dataset and transferring a proportion of parameters, where the bound depends not only the strength of signal and noise, but also the impact of sample size. Furthermore, the theory establishes sufficient condition for determining when transfer is beneficial or harmful, i.e., negative transfer. Several numerical and realistic experiments are included as an assist.

Due to the concerns on novelty and the connections between experiments and the theoretical setup, the reviewer is moderately inclined to reject the submission.

Critical strengths

The reviewer highly appreciates the detailed analyses present in this paper, and the effort in conducting an array of experiments, not only on authors' toy model, but also trying to connect to more realistic practices.

Critical weaknesses

Novelty

To the best of the reviewer's knowledge, this work unfortunately may not be the first theoretical work on parameter transfers (line 52-53).

The reviewer found that the analyses appeared to be shared beyond the skeleton of the analyses presented by Meng et al., 2024 and Kou et al., 2023. Moreover, from other communities, feature learning dynamics on transfer learning has been analytically studied in deep linear networks (e.g., Lampinen and Ganguli, 2019; a more recent one, see Tahir et al., 2025). Moreover, a precise asymptotic analysis on perceptron models also revealed similar phenomena (Dhifallah and Lu, 2021). All of these papers include the discussion of negative transfer.

Therefore, the reviewer recommends the authors better contextualize their work within the theoretical research of the topic and highlight the key contributions. Specifically, it would be helpful if the authors could more explicitly indicate the theoretical advancements compared to existing literature (e.g., Meng et al. and Kou et al.) and emphasize the novel insights offered by this model that cannot be captured by perceptron-based models or deep linear networks.

Experiment

The reviewer greatly appreciates authors' efforts into verifying the theory in simulations and connecting it to more common practice. However, the reviewer also wants to raise the worry about how these experiments can support the theory:

First, it seems the parameters chosen for numerical experiments may not satisfy the conditions of 4.1. For example, the dimension d=2000d=2000 is noticeably smaller than n2=(100)2n^2 = (100)^2 (i.e., the first two settings, while the other settings used a larger nn), which further violates the condition 1. Likewise, the number of filters, the signal norm, the learning rates, the initial noise, and the required Γ\Gamma in Proposition 4.4 seem all fall short of the required conditions of the theory. Thus, the reviewer can only conclude that the experiments can serve as qualitative support for the predicted trend rather than the theorems. Therefore, the reviewer was wondering if the author could further explain, justify, or try to run the experiment on a smaller scale to support the main theorem.

Second, the reviewer interprets the theorem as evaluating not only accuracy but also the phase transition from good to bad transfers. Please correct the following if it is incorrect: The reviewer's reading of the second as well as the third statement in Theorem 4.2 is the transition depends only on dimension dd and the term (α2N12∥u∥24σp4+N22∥u+v2∥24σp4)/(α2σp2N1σp2+N2)=:R\left(\frac{\alpha^2 N_1^2 \|\mathbf{u}\|_2^4}{\sigma_p^4} + \frac{N_2^2 \|\mathbf{u}+\mathbf{v}_2\|_2^4}{\sigma_p^4}\right) \bigg/ \left(\frac{\alpha^2 \sigma_p^2 N_1}{\sigma_p^2} + N_2\right) =: R. If that is the case, could the authors try different configurations of the denominator and numerator with different dd and proportion α\alphas and plot the R/dR/d values when the error reaches the same level? This will show whether the R/dR/d ratio stays the same and how close the actual error is to it. Furthermore, could the authors further plot the regressor of R/d to see if the log error is linear? The speed of decay may also indicate how loose the bound is.

Furthermore, the reviewer can understand the potential theoretical convenience for considering correlated signals and/or correlated noise. But as authors hinted (line 159-161), is it possible to numerically verify it in a controlled setting and see, for example, whether negatively correlated signals can impose a stronger negative transfer effect? It would also be helpful to explicitly test on correlated noise (in different degrees) with the other signal direction. In addition, from the formula to the reviewer, it seems in different regimes, practitioners should either do complete parameter transfer or not to do it at all (which seems similar to the results in Dhifallah and Lu, 2021). Could the authors further experimentally verify this idea (and possibly include the correlated scenario, because it could be the case that there is an optimal non-full transfer)?

Finally, regarding to the ViT experiments, the paper documented that "… and its 9th, 10th, and 11th layers are selected as inherited parameters" (line 2537 and 2571). But to the best of reviewer's knowledge, that is quite different to the common practice in transfer learning tasks, where practitioners usually just replace the last layers and fine-tune the model on the downstream task (e.g., Task 2 in the paper). This practice is built on the belief that the early layers learned the common features, while the last layers are related to downstream tasks. So, the reviewer would like to know authors' justification of this decision and how much does this setting link to the common practice. Or, alternatively, the reviewer was wondering whether authors here assumed the feature / knowledge representations distributed equally in the network.

Writing

Unfortunately, the reviewer found the paper difficult to read. Potentially due to the page limit, the proof sketch in the main paper lacks a clear link between the training dynamics of the weights and the two systems, as well as how it links to the final theorems. For example, it is unclear why we need to consider a system in which the solutions are given by the variables xˉtA\bar x_t^A and x‾tA\underline x_t^A. The descriptions of bˉtA,c‾tA,cˉtA,c‾tA\bar b^A_t, \underline c^A_t, \bar c^A_t, \underline c_t^A are also vague in the main text.

The authors provided technical motivations and generic explanations for the conditions and theorems. However, given the number of terms, a more precise technical interpretations of the results would be helpful. These interpretations are welcome to include the reasons for and consequences of the order of some key terms in the bound. Importantly, could the author provide insight into the constant terms, such as C1C_1 and C3C_3, in Theorem 4.2, including if (and how) they are connected?

Questions for authors

Please see my comments in the Weakness part.

The reviewer has the following minor points:

  1. Could the authors confirm whether there is a typo in line 8 of Algorithm 1 (i.e. T∗∗T^{**} instead of T∗T^{*})?
  2. The authors defined the shorthand notation of poly(… )poly(\dots) as a positive power of variables in the parentheses. So, for example, should the reader read T∗T^{*} as =η−1poly(n,d,1/ϵ,m)= \eta^{-1} poly(n, d, 1/\epsilon, m)?
  3. Likewise, in lines 229–230, do the authors mean width ww or dimension dd by width d? Should the reviewer interpret it as the latter (boldfaced)?
  4. With regard to the first point of Theorem 4.2, are the authors referring to the training error for Task 2? LSL_S itself seems undefined in this paper (it can be found in the paper that the authors cited, i.e., Kou et al. 2023). The reviewer found it confusing because, although the proof (F.12) starts with index 2, it quickly switches to σp,12\sigma^2_{p, 1}. The previous referred Lemma seems is also exclusively for task 2, so the reviewer found it difficult to follow.
  5. Similarly, for the third point, according to Kou et al., the reviewer assumes it corresponds to a different constant (presumably C3′C_{3}', but please correct me if I'm wrong).
  6. Regarding the second point in Proposition 4.4, is it α2\alpha^2 instead of α\alpha? If the previous point is correct, should there be 1/C3′1/C'_{3} instead?
  7. Could the authors discuss why CoNLL2003 has a big drop compared to the other two datasets? Is it because using a different metric (F1 reported rather than accuracy)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.