- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies fine-tuning in a Gaussian teacher-student setting for two-layer ReLU networks, assuming that pretraining has recovered m teacher features and that the downstream task introduces one additional feature. With an infinitely faster output layer and a vanishing-scale additional hidden neuron, the authors derive limiting dynamics that preserve the pretrained hidden directions while learning the new direction. Under orthonormal teacher features and a dimensional condition scaling quadratically with m, they prove recovery from O(d) fine-tuning samples, without an additional factor depending on the number of pretrained features. Synthetic experiments compare this approach with full fine-tuning, linear probing, LP-FT, and L2-SP.
Critical strengths
The main statistical result combines recovery of a new nonlinear feature with preservation of an existing representation, using a sample bound without an additional dependence on the number of pretrained features. The analysis also identifies how the combination of rapid output-layer fitting and the small norm of an additional hidden neuron separates the directional dynamics of new and pretrained features. This gives a concrete explanation of selective feature learning in the idealized setting considered.
Critical weaknesses
-
The novelty relative to prior work is insufficiently established. The broad message—that fine-tuning can learn new task-specific features while benefiting from pretrained representations—already appears in closely related theoretical work. In particular, Dayi & Chen (2024) establish non-kernel feature learning and advantages over learning from scratch, while Lee et al. (2022), absent from the related work, analyze the reuse of specialized neurons versus the activation of previously inactive ones during sequential learning. Although neither directly subsumes the present theorem, the paper should more clearly distinguish its contribution in implicit feature preservation and width-independent sample complexity from these existing results. Moreover, its characterization of Anguita et al. (2026) as lacking a statistical analysis of generalization is inaccurate: that work explicitly derives analytical expressions for generalization error, albeit for diagonal linear networks.
-
The theoretical guarantees rely on restrictive assumptions and limiting dynamics, leaving their practical applicability unclear. The analysis assumes that all teacher features are exactly orthonormal, the pretrained model has recovered every shared feature perfectly, and only one new feature remains to be learned. The main recovery guarantee further requires and concerns limiting dynamics with an infinitely fast output layer and a vanishing-scale additional neuron. These conditions leave open whether the favorable behavior persists with correlated features, imperfect pretrained representations, and finite learning-rate ratios. Although the experiments provide evidence beyond some assumptions, they do not establish robustness to substantial feature correlations, and the claimed relaxation of orthogonality is not accompanied by a quantitative theorem. Consequently, the sample guarantee remains confined to a highly idealized setting.
Questions for authors
-
Could you clarify the precise novelty relative to Dayi & Chen (2024), separating the contribution of implicit feature preservation from that of the sample bound, and address the related node-reuse/node-activation analysis of Lee et al. (2022)? Please also clarify the characterization of Anguita et al. (2026), which explicitly analyzes generalization error in diagonal linear networks.
-
What quantitative guarantees, if any, remain when teacher features have nonzero correlations, the pretrained features are imperfect, or the learning-rate ratio and additional-neuron scale are finite? In particular, what explicit angular condition would replace orthogonality, and how would the sample requirement depend on it?
2 comments
Thank you for the detailed response and for clarifying the distinctions from prior work (Dayi & Chen, 2024; Lee et al., 2022; Anguita et al., 2026). While the clarifications are helpful, my main concern regarding the narrowness of the theoretical setting remains: the formal guarantees still rely on exact feature orthogonality and only handle a single new fine-tuning feature, while the extension to nearly orthogonal features () is discussed only heuristically without a formal proof, and feature distortion under correlation remains open. Therefore, I will maintain my original score.
Authors Rebuttal
We thank the reviewer for their comments and questions, which we will take into account when revising the paper. We address them below.
W1. The novelty relative to prior work is insufficiently established.
We thank the reviewer for pointing out these related works. We clarify the main differences below and will expand the related-work discussion in the revised version.
Dayi & Chen (2024) consider a related pre-training/fine-tuning task, but study parameter-efficient fine-tuning: the pre-trained parameters are frozen and only a rank-one component is optimized. By contrast, we study full fine-tuning, where all parameters are allowed to evolve. The preservation of pre-trained hidden features is an outcome of our two-timescale dynamics rather than an algorithmic constraint. Importantly, our output weights are adapted during fine-tuning, which makes the resulting dynamics substantially richer (and more novel wrt existing literature) than learning a single additional neuron.
Lee et al. (2022). We thank the reviewer for bringing this work to our attention. They consider a single pre-trained feature and a fine-tuning teacher consisting of a new feature. Using statistical-physics arguments, they derive a limiting ODE as d→∞, without a rigorous justification of this limit. Their conclusions are then primarily based on numerical investigations of this ODE, rather than on rigorous mathematical statements about the resulting dynamics. Interestingly, they observe phenomena related to ours for uncorrelated features, as well as distortion of the pre-trained feature when the two features are correlated. Whether such distortion occurs in our setting, where the pre-trained features remain necessary for fitting the fine-tuning task, is an interesting open question.
Anguita et al. (2026). Their generalization analysis similarly relies on statistical-physics heuristics (the replica assumption) to derive the asymptotic generalization error as d→∞, which is then evaluated numerically. In contrast, our results provide rigorous mathematical guarantees on the training dynamics and finite-sample complexity. We will clarify this distinction in the revised version.
W2. The theoretical guarantees rely on restrictive assumptions and limiting dynamics, leaving their practical applicability unclear.
As discussed in our response to Reviewer dnMs, extending our analysis to multiple additional fine-tuning features would already require addressing substantial difficulties that arise when learning multiple ReLU features from scratch. A complete characterization of the dynamics for simultaneously learning multiple unknown ReLU features would itself constitute a significant advance; extending such an analysis to our setting with both multiple pre-trained and multiple new fine-tuning features is therefore beyond the scope of the present work.
Regarding Assumption 2 (orthogonality of the features), we first emphasize that this assumption is not imposed in our experiments: the pre-training features are instead sampled independently from a Gaussian distribution and are therefore only approximately orthogonal in high dimension. A similar orthogonality assumption is also made by Dayi & Chen in their theoretical analysis. More importantly, exact orthogonality is not essential to our analysis. Our arguments can be extended to nearly orthogonal features, for instance under pairwise correlations ⟨wi⋆,wj⋆⟩=O(1/m). Under such a condition, we expect the same convergence rates and no change in the order of the required sample complexity. The main additional work consists in controlling perturbations of the population dynamics; the empirical concentration arguments, which determine the sample-complexity scaling, are essentially unchanged.
We chose to state the results under exact orthogonality to keep an already technical analysis as transparent as possible, since we do not expect the nearly orthogonal extension to provide additional conceptual insight. We will clarify more precisely in the revised version that exact orthogonality is not essential and explain how the analysis extends to the nearly orthogonal case.