acceptodds
Under review as a conference paper at ICLR 2027

Two-Timescale Fine-tuning Provably Learns New Features for Two-Layer ReLU Networks

Abstract

Fine-tuning pre-trained models on specialized tasks with scarce data is central to modern deep learning. Despite its empirical success, theoretical understanding of fine-tuning remains limited. We introduce a Gaussian multi-index setting to study fine-tuning from pre-trained weights, where the teacher network has features, of which are learned during pre-training and one of which must be learned during fine-tuning. For two-layer ReLU networks, we show that two-timescale training, i.e., updating the outer weights infinitely faster than the hidden ones, learns the new task-specific feature while preserving the pre-trained ones in the model representation. Moreover, only fine-tuning samples are required for this recovery, independently of the number of pre-trained features. In contrast, with random initialization, the same number of samples is insufficient to recover the target parameters. Our results therefore demonstrate that pre-training can induce an implicit bias with a clear statistical advantage over random initialization, enabling feature learning from scarce fine-tuning data.

Primary area:
learning theory
Keywords:
Finetuning, Two-Layer ReLU networks, Multi-index Gaussian models, Two-timescale, Feature Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bXuaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies fine-tuning in a Gaussian teacher-student setting for two-layer ReLU networks, assuming that pretraining has recovered m teacher features and that the downstream task introduces one additional feature. With an infinitely faster output layer and a vanishing-scale additional hidden neuron, the authors derive limiting dynamics that preserve the pretrained hidden directions while learning the new direction. Under orthonormal teacher features and a dimensional condition scaling quadratically with m, they prove recovery from O(d) fine-tuning samples, without an additional factor depending on the number of pretrained features. Synthetic experiments compare this approach with full fine-tuning, linear probing, LP-FT, and L2-SP.

Critical strengths

The main statistical result combines recovery of a new nonlinear feature with preservation of an existing representation, using a sample bound without an additional dependence on the number of pretrained features. The analysis also identifies how the combination of rapid output-layer fitting and the small norm of an additional hidden neuron separates the directional dynamics of new and pretrained features. This gives a concrete explanation of selective feature learning in the idealized setting considered.

Critical weaknesses
  1. The novelty relative to prior work is insufficiently established. The broad message—that fine-tuning can learn new task-specific features while benefiting from pretrained representations—already appears in closely related theoretical work. In particular, Dayi & Chen (2024) establish non-kernel feature learning and advantages over learning from scratch, while Lee et al. (2022), absent from the related work, analyze the reuse of specialized neurons versus the activation of previously inactive ones during sequential learning. Although neither directly subsumes the present theorem, the paper should more clearly distinguish its contribution in implicit feature preservation and width-independent sample complexity from these existing results. Moreover, its characterization of Anguita et al. (2026) as lacking a statistical analysis of generalization is inaccurate: that work explicitly derives analytical expressions for generalization error, albeit for diagonal linear networks.

  2. The theoretical guarantees rely on restrictive assumptions and limiting dynamics, leaving their practical applicability unclear. The analysis assumes that all teacher features are exactly orthonormal, the pretrained model has recovered every shared feature perfectly, and only one new feature remains to be learned. The main recovery guarantee further requires d≳m2log⁡(1/δ)d\gtrsim m^2\log(1/\delta) and concerns limiting dynamics with an infinitely fast output layer and a vanishing-scale additional neuron. These conditions leave open whether the favorable behavior persists with correlated features, imperfect pretrained representations, and finite learning-rate ratios. Although the experiments provide evidence beyond some assumptions, they do not establish robustness to substantial feature correlations, and the claimed relaxation of orthogonality is not accompanied by a quantitative theorem. Consequently, the O(d)O(d) sample guarantee remains confined to a highly idealized setting.

Questions for authors
  1. Could you clarify the precise novelty relative to Dayi & Chen (2024), separating the contribution of implicit feature preservation from that of the sample bound, and address the related node-reuse/node-activation analysis of Lee et al. (2022)? Please also clarify the characterization of Anguita et al. (2026), which explicitly analyzes generalization error in diagonal linear networks.

  2. What quantitative guarantees, if any, remain when teacher features have nonzero correlations, the pretrained features are imperfect, or the learning-rate ratio and additional-neuron scale are finite? In particular, what explicit angular condition would replace orthogonality, and how would the sample requirement depend on it?

2 comments
AuthorsOpenReview ↗

Authors Rebuttal

We thank the reviewer for their comments and questions, which we will take into account when revising the paper. We address them below.

W1. The novelty relative to prior work is insufficiently established.

We thank the reviewer for pointing out these related works. We clarify the main differences below and will expand the related-work discussion in the revised version.

Dayi & Chen (2024) consider a related pre-training/fine-tuning task, but study parameter-efficient fine-tuning: the pre-trained parameters are frozen and only a rank-one component is optimized. By contrast, we study full fine-tuning, where all parameters are allowed to evolve. The preservation of pre-trained hidden features is an outcome of our two-timescale dynamics rather than an algorithmic constraint. Importantly, our output weights are adapted during fine-tuning, which makes the resulting dynamics substantially richer (and more novel wrt existing literature) than learning a single additional neuron.

Lee et al. (2022). We thank the reviewer for bringing this work to our attention. They consider a single pre-trained feature and a fine-tuning teacher consisting of a new feature. Using statistical-physics arguments, they derive a limiting ODE as d→∞d\to\infty, without a rigorous justification of this limit. Their conclusions are then primarily based on numerical investigations of this ODE, rather than on rigorous mathematical statements about the resulting dynamics. Interestingly, they observe phenomena related to ours for uncorrelated features, as well as distortion of the pre-trained feature when the two features are correlated. Whether such distortion occurs in our setting, where the pre-trained features remain necessary for fitting the fine-tuning task, is an interesting open question.

Anguita et al. (2026). Their generalization analysis similarly relies on statistical-physics heuristics (the replica assumption) to derive the asymptotic generalization error as d→∞d\to\infty, which is then evaluated numerically. In contrast, our results provide rigorous mathematical guarantees on the training dynamics and finite-sample complexity. We will clarify this distinction in the revised version.

W2. The theoretical guarantees rely on restrictive assumptions and limiting dynamics, leaving their practical applicability unclear.

As discussed in our response to Reviewer dnMs, extending our analysis to multiple additional fine-tuning features would already require addressing substantial difficulties that arise when learning multiple ReLU features from scratch. A complete characterization of the dynamics for simultaneously learning multiple unknown ReLU features would itself constitute a significant advance; extending such an analysis to our setting with both multiple pre-trained and multiple new fine-tuning features is therefore beyond the scope of the present work.

Regarding Assumption 2 (orthogonality of the features), we first emphasize that this assumption is not imposed in our experiments: the pre-training features are instead sampled independently from a Gaussian distribution and are therefore only approximately orthogonal in high dimension. A similar orthogonality assumption is also made by Dayi & Chen in their theoretical analysis. More importantly, exact orthogonality is not essential to our analysis. Our arguments can be extended to nearly orthogonal features, for instance under pairwise correlations ∣⟨wi⋆,wj⋆⟩∣=O(1/m)\left|\langle w_i^\star,w_j^\star\rangle\right|=O(1/m). Under such a condition, we expect the same convergence rates and no change in the order of the required sample complexity. The main additional work consists in controlling perturbations of the population dynamics; the empirical concentration arguments, which determine the sample-complexity scaling, are essentially unchanged.

We chose to state the results under exact orthogonality to keep an already technical analysis as transparent as possible, since we do not expect the nearly orthogonal extension to provide additional conceptual insight. We will clarify more precisely in the revised version that exact orthogonality is not essential and explain how the analysis extends to the nearly orthogonal case.

Thank you for the detailed response and for clarifying the distinctions from prior work (Dayi & Chen, 2024; Lee et al., 2022; Anguita et al., 2026). While the clarifications are helpful, my main concern regarding the narrowness of the theoretical setting remains: the formal guarantees still rely on exact feature orthogonality and only handle a single new fine-tuning feature, while the extension to nearly orthogonal features (O(1/m)O(1/m)) is discussed only heuristically without a formal proof, and feature distortion under correlation remains open. Therefore, I will maintain my original score.

Reviewer_5jnUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies fine-tuning in a teacher–student setting with two-layer ReLU networks and Gaussian inputs. The downstream teacher contains m+1m+1 features, and the student is initialized with exact knowledge of mm teacher features (supposedly recovered after pre-training) and one additional neuron of small norm. The authors analyze two-timescale gradient-flow dynamics in which the output weights are continuously optimized while the hidden weights evolve more slowly. In the small-initialization limit, the dynamics learn the missing feature while preserving the pretrained ones. By assuming orthonormal teacher features and an input dimension sufficiently large relative to the number of pretrained features, the authors prove convergence of the fine-tuning stage for both population and empirical losses, with only O(d)\mathcal O(d) fine-tuning samples required for recovery. This establishes a statistical advantage from exploiting pretrained features compared with learning the full teacher from scratch. Numerical comparisons with standard fine-tuning baselines show that two-timescale training achieves low test loss while largely preserving the pretrained features.

Critical strengths

The paper is clearly written and the derivations are easy to follow. I particularly appreciate the authors dedicated discussion of the limitations, including those arising from the assumptions on the pretrained initialization. The sample-complexity result is another strength: under the authors assumptions, learning the missing feature requires O(d)\mathcal O(d) downstream samples, compared with the O(md)\mathcal O(md) samples required to learn the full teacher from scratch.

Critical weaknesses
  1. In this setting there is no intrinsic tension between preserving the pretrained features and learning the downstream task: the old features remain useful, and an additional neuron is available to learn the new one. I think this limits what we can conclude about fine-tuning more generally, where adapting to a new task may require modifying or even discarding some of the pretrained features. I would appreciate a discussion of this point: to what extent does the proposed mechanism help us understand settings where preserving the pretrained representation is not necessarily the best strategy for downstream performance?

  2. I find the assumed state of the network after pretraining rather restrictive. With an overparameterized student, there need not be a one-to-one correspondence between student and teacher neurons: several student neurons could specialize to the same teacher feature, even when the pretraining loss is very small. With limited data or incomplete pretraining, the learned features may also only approximately align with the teacher features. Moreover, low pretraining loss does not guarantee the assumed separation of neuron norms, and a neuron contributing little to the output does not necessarily have a small hidden-weight norm. I appreciate that the authors acknowledge these limitations and include an experiment with actual pretraining, but I would like to better understand how well the assumed initialization describes the resulting network. Could the authors report the neuron norms and teacher–student overlaps after pretraining? It would also be useful to understand how sensitive the results are to imperfect feature recovery and features represented by multiple neurons.

  3. I would also like to better understand how much the results depend on the idealized optimization dynamics. I appreciate that the experiments use minibatch SGD, but how sensitive are the results to the learning-rate ratio, step size, and batch size? Moreover, the mechanism relies on norm-dependent directional learning rates, which may change under other optimizers. What would the authors expect to happen with adaptive or preconditioned optimizers?

At the moment, I find the scope of the paper too limited, mainly because it remains unclear to me how broadly the conclusions apply beyond the idealized setting. However, I am more than willing to reconsider my score if the authors address these concerns and clarify the broader implications of their results.

Questions for authors

See above.

3 comments
AuthorsOpenReview ↗

Authors rebuttal (1/2)

We thank the reviewer for their comments and questions, which we will take into account when revising the paper. We address them below.

W1. In this setting there is no intrinsic tension between preserving the pretrained features and learning the downstream task.

We agree that, for the particular downstream task considered in our theoretical analysis, the pre-trained features do not need to be modified. Crucially, however, they are not intrinsically frozen: their preservation is a consequence of the first timescale analyzed in Proposition 1. On this accelerated timescale, the void feature evolves, while changes in the pre-trained features occur only on a slower timescale and are therefore asymptotically negligible.

On the original, slower timescale, these pre-trained features can subsequently evolve. Thus, for a downstream task requiring both the acquisition of new features and the adaptation of pre-trained ones, one could expect a two-stage dynamics: first, the missing features are learned on the fast timescale while the pre-trained representation is preserved; then, on a slower timescale, the pre-trained features themselves adapt to the downstream task. In our present setting, the first stage already perfectly fits the downstream task, so there is no subsequent gradient driving such adaptation, and we therefore do not analyze this second stage.

We believe that studying this richer multi-timescale dynamics would be very interesting, but would require a substantially more involved analysis and is beyond the scope of the present work.

W2. I find the assumed state of the network after pretraining rather restrictive.

As discussed in the paper, our initialization is an idealization of the state obtained after sufficiently long pre-training with abundant data. In practice, pre-training an overparameterized network can lead to two additional phenomena. First, several neurons may specialize to the same pre-trained feature. This does not fundamentally affect our analysis: if several neurons represent the same wi⋆w_i^\star, the corresponding output weight can be distributed among them without changing the represented function or the qualitative dynamics. Second, the network may contain several void features. We already discuss this case as an interesting direction for future work.

After sufficiently successful pre-training, a void feature should contribute very little to the network output. This can occur through a small output weight, a small hidden-weight norm, or both. In our experiments, we typically observe both weights becoming small, which is due to the approximate balancedness property and is further reinforced by weight decay. We have added Figure 5 in Appendix A, showing the features of the network at the end of pre-training. It illustrates both phenomena described above: multiple neurons specialize to individual pre-training features, while some neurons remain effectively void, with markedly smaller norms. Interestingly, with sufficiently long pre-training, nearly all neurons eventually align with pre-trained features, suggesting a possible collapse of the available representation and reducing the number of void features available for subsequent fine-tuning.

AuthorsOpenReview ↗

Authors rebuttal (2/2)

W3. I would also like to better understand how much the results depend on the idealized optimization dynamics.

As with most optimization procedures, performance is somewhat sensitive to the learning-rate ratio and step size, as well as to the hyperparameters used during pre-training. In our experiments, batch size appears to have a much smaller effect. Since the primary contribution of this work is theoretical, we do not include an exhaustive hyperparameter study. Interestingly, across our experiments, we found two-timescale fine-tuning to be more robust to these hyperparameter choices than the competing methods. In the paper, we report the best hyperparameter configuration for each method to ensure a fair comparison, which does not reflect this additional robustness.

Regarding adaptive optimizers, we repeated the experiment of Appendix A.4 using Adam instead of SGD and obtained qualitatively similar results. We believe there is also a simple intuition for why the mechanism should persist. The 1/∥w∥21/\|w\|^2 scaling of the directional dynamics in Equation (3) can be understood as arising from two factors. One factor 1/∥w∥1/\|w\| comes from the homogeneity of the architecture and the corresponding scaling of the gradient; this factor is specific to scale-homogeneous gradient-based updates and is not expected to persist with Adam. The second factor 1/∥w∥1/\|w\|, however, is purely geometric: an update of order one to ww induces an update of order 1/∥w∥1/\|w\| to its direction. This effect is not specific to SGD.

We therefore expect the same qualitative separation of directional timescales to persist with adaptive optimizers such as Adam, although with a weaker 1/∥w∥1/\|w\|, rather than 1/∥w∥21/\|w\|^2, scaling. Consequently, smaller initialization scales may be required to approach the limiting dynamics studied in Proposition 1. While this argument is only heuristic and our theory does not currently cover Adam, the additional experiment supports this prediction.

We will be happy to include this additional experiment in the revised version.

How broadly do the conclusions apply beyond the idealized setting?

Our main contribution is theoretical, and establishing the full scope of these conclusions in realistic large-scale settings would constitute a substantial empirical study in its own right. Nevertheless, our experiments with actual pre-training already relax several assumptions of the theoretical model simultaneously and recover the same qualitative behavior, suggesting that these assumptions are not all necessary in practice.

We would be happy to include additional experiments in the revised version if the reviewer has specific suggestions that they believe would help clarify the robustness of the phenomenon.

Thank you for the clarifications and additional diagnostics. I appreciate the experiement with Adam. However, my main concerns about the scope and implications of the results remain. Also, the obeservation that for longer pretraining "nearly all neurons eventually align with pre-trained features, suggesting a possible collapse of the available representation" raises an unresolved question about when the initialization structure required by this mechanism actually persist.

I remain unconvinced that the results provide sufficiently broad insight beyond this favorable setting. I therefore maintain my score.

Reviewer_dnMsOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper analyzes fine-tuning of a two-layer ReLU network in a Gaussian multi-index teacher–student setting. Pretraining is idealized as having exactly recovered m teacher features, while the remaining “void” feature starts with small norm and a random direction. Under a two-timescale gradient-flow limit, the output weights adjust instantaneously and the hidden weights evolve slowly. The authors show that, in the limit, only the new feature moves while the pretrained features stay fixed. They characterize the limiting dynamics, prove linear convergence on the population loss, and establish an empirical-loss guarantee with n ≳ d samples, independent of m, versus Ω(md) for learning all features from scratch. Toy experiments compare full fine-tuning, two-timescale, LP-FT, L2-SP, and linear probing.

Critical strengths

1.Clean theoretical mechanism. The two-timescale limit isolates the new feature while freezing pretrained features. This is consequential because it gives a precise optimization-level explanation for a widely used fine-tuning heuristic. 2.End-to-end statistical guarantee. Combining the limiting dynamics with concentration to obtain n ≳ d independent of m is the paper’s main contribution. It makes the statistical advantage of fine-tuning concrete rather than heuristic.

Critical weaknesses

1.Generality is limited by strong assumptions. Orthogonal features, exact pretrained initialization, a vanishing void-feature scale, and a single new feature are far from practical fine-tuning. This matters because the headline sample-complexity claim may not survive realistic feature correlations or multiple new features. 2.Experimental validation is toy-scale. The experiments illustrate the theory but do not establish practical relevance at realistic scale; the real-pretraining appendix is useful but limited.

Questions for authors

1.How sensitive are the convergence rates and sample complexity to violation of orthogonality and to finite learning-rate ratios? 2.Is the Ω(md) from-scratch lower bound formal, or only an information-theoretic intuition? 3.Can the analysis be extended to multiple new features, or does the mechanism break down?

3 comments
AuthorsOpenReview ↗

Authors rebuttal (1/2)

We thank the reviewer for their comments and questions, which will be taken into account in the rewriting of this work. We answer them below.

W1. Generality is limited by strong assumptions.

Strong structural assumptions are common in theoretical analyses of neural-network dynamics, as relaxing them is often beyond current techniques. We discuss the role of each assumption and possible relaxations in the Discussion section. We address the reviewer's specific concerns below.

non-orthogonal features

Proposition 1, which establishes the two-timescale limit with vanishing void features, does not require orthogonality. Orthogonality is used only to analyze the resulting limiting dynamics. Our analysis could be extended to nearly orthogonal features, e.g., under pairwise correlations ∣⟨wi⋆,wj⋆⟩∣=O(1/m)|\langle w_i^\star,w_j^\star\rangle|=O(1/m), without changing the order of the sample complexity. We chose the orthogonal setting for clarity, as we do not expect this extension to provide substantially new theoretical insight. Larger correlations, however, can lead to qualitatively different dynamics, as illustrated by [1] in a slightly different setting, suggested by Reviewer bXua.

multiple new features

A complete global analysis of learning multiple ReLU features remains largely out of reach even without pre-training. Extending our results to multiple new features would therefore require resolving difficulties already present in the training-from-scratch setting, in addition to those specific to fine-tuning.

vanishing void-feature scale

Our analysis indeed focuses on the limit α→0\alpha\to0. Characterizing the dependence on finite α\alpha appears substantially more challenging, and we leave this question for future work.

Overall, while our assumptions simplify the setting, the resulting problem remains technically very challenging and allows us to obtain a detailed characterization of both the training dynamics and sample complexity. Without these assumptions, such a detailed analysis would be beyond the reach of current theoretical techniques. Moreover, the real pre-training experiment in the appendix suggests that the qualitative conclusions persist when these assumptions are simultaneously relaxed.

W2. Experimental validation is toy-scale

This is a primarily theoretical work, and our goal is to isolate and rigorously characterize a mechanism by which fine-tuning can learn a new feature while preserving pre-trained ones. Large-scale empirical validation would constitute a substantial additional contribution beyond this scope.

Importantly, using different learning rates across layers—the practical counterpart of our two-timescale idealization—has already been extensively studied empirically; see, e.g., Howard & Ruder (2018) and [2,3]. Our experiments instead focus on illustrating the specific mechanism predicted by the theory.

Q1. How sensitive are the convergence rates and sample complexity to violation of orthogonality and to finite learning-rate ratios?

For nearly orthogonal features, e.g., ∣⟨wi⋆,wj⋆⟩∣=O(1/m)|\langle w_i^\star,w_j^\star\rangle|=O(1/m), we expect our analysis to extend with similar convergence rates and without changing the order of the sample complexity. For larger correlations, however, the dynamics can change qualitatively; see [1].

Our current analysis does not characterize finite learning-rate ratios. The two-timescale limit should be viewed as an idealization of discriminative learning rates, and quantifying the approximation for finite ratios is an interesting direction for future work.

Q2. Is the Ω(md) from-scratch lower bound formal, or only an information-theoretic intuition?

With non-zero observation noise, standard information theoretic arguments yield an Ω(md)\Omega(md) sample-complexity lower bound for recovering mm-generic dd-dimensional features; see, e.g., [4].

While our main results consider noiseless labels, the early-stage arguments responsible for learning the new feature are robust to observation noise. In particular, they yield recovery of the new feature up to error ε\varepsilon with O(d/ε2)O(d/\varepsilon^2) samples. Noiselessness is only used in the late stage to establish exact convergence to the ground truth.

Thus, in the noisy setting, this comparison can be made formal: recovering all mm features from scratch requires Ω(md)\Omega(md) samples for constant-accuracy recovery, whereas two-timescale fine-tuning requires only O(d)O(d) samples.

In the noiseless setting, [5] provides a related result: local identifiability of a ReLU network requires at least Ω(md)\Omega(md) samples in our setting. While identifiability is distinct from the feature-recovery criterion considered here, this result provides further support for the same Ω(md)\Omega(md) scaling.

AuthorsOpenReview ↗

Authors rebuttal (2/2)

Q3. Can the analysis be extended to multiple new features, or does the mechanism break down?

Current global analyses of learning ReLU features from scratch are largely restricted to a single unknown neuron, with only a few exceptions in more specific settings, such as linear teachers or local-convergence analyses. A complete characterization of the dynamics for simultaneously learning multiple unknown ReLU features would already represent a significant advance in the training-from-scratch setting. Extending such an analysis to our more challenging fine-tuning setting, with both multiple pre-trained and multiple new features, is therefore beyond the scope of the present work and constitutes an interesting but challenging direction for future research.

References

[1] Lee et al., ICML 2022, "Maslow's hammer for catastrophic forgetting: Node re-use vs node activation."

[2] Shen et al., AAAI 2021, “Partial Is Better Than All: Revisiting Fine-tuning Strategy for Few-shot Learning.”

[3] Ro & Choi, AAAI 2021, “AutoLR: Layer-Wise Pruning and Auto-Tuning of Learning Rates in Fine-Tuning of Deep Networks.”

[4] Loh, Entropy 2017, “On Lower Bounds for Statistical Learning Theory”.

[5] Bona-Pellissier et al., NeurIPS 2022, “Local Identifiability of Deep ReLU Neural Networks: the Theory”.

Thank you for the clarifications. I appreciate the technical challenges involved in relaxing the assumptions. However, limited generality also indeed reduce its contribution. After considering the response and the discussion, I have decided to maintain my score.

Comments on the paper

1 comment
Area_Chair_KUT2OpenReview ↗

Discussion phase coming to an end

Dear reviewers,

As we are approaching the deadline of the discussion phase, please read the authors’ responses and share your thoughts in the discussion as soon as possible. In particular, please clarify whether the responses have addressed your main concerns. Please also update your review and scores if appropriate.

Your timely input will help us reach a well-informed decision. Thank you for your time and effort!

Best regards, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.