- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This paper addresses the fine-tuning paradigm, where a large pretrained foundation model is trained further on some downstream dataset. Specifically, the work aims to characterize how the pretrained initialization benefits the fine-tuning process. The authors consider sparse linear regression and two-layer diagonal linear neural networks. They consider a setup where the model is initialized such that it already has partial information about the target vector, for example if some of the parameters already correctly match the target parameters. They demonstrate that gradient descent will exploit these similarities to converge with improved sample complexity.
Critical strengths
The overall problem is interesting and important.
Critical weaknesses
- The settings examined in the paper are highly constrained and not well motivated. Why consider sparse linear regression, and not general linear regression? Why consider diagonal linear neural networks, and not general linear neural networks? These limit the usefulness of the theoretical results.
- The analysis does not really capture core issues of the fine-tuning framework. First, it does not model the pretraining process, taking the pretrained model as given. It assumes the model is initialized such that it already correctly identifies relevant coordinates, i.e. correctly models some of the data, which is a strong assumption. A core concern of fine-tuning is the distribution shift between the pretraining data and fine-tuning data, which this analysis does not capture. The analysis also does not consider other concerns of fine-tuning, such as parameter-efficient fine-tuning, self-supervised pretraining vs. supervised fine-tuning, or large nonconvex models with suboptimal stationary points.
- Ultimately, the conclusion of this work may be summarized as, if the model is initialized at a point where it is close to the desired solution, then it will converge with fewer samples to the desired solution. This seems somewhat circular and does not reveal new insights.
Additional weakness
- The use of the word "support" is somewhat confusing. The paper would benefit from defining it in one line. I understand it is to mean the indices where the vector is non-zero (as opposed to a support vector).
- Some terminology seems unclear and nonstandard. For example, what is a "contaminated face"?
- The writing is dense and convoluted, making it difficult to discern the main contribution of this work. An example of this is the following sentence:
"This suggests a statistical gain: the ambient dimensional logarithmic cost should only be incurred for coordinates whose membership in the target support is unknown, while the cost of estimating the coefficients on all s coordinates remains."
Another example is:
"We first consider an ideal stopping rule which stops the first time, if finite, is included in the estimated support; and returns the corresponding saddle. This rule is not available in practice, but isolates the main statistical question: does the trajectory recover the remaining signal before proposing a new null coordinate?"
This work would benefit from rewriting with the goal of making it understandable to a human reader.
Questions for authors
See questions from the above section.
4 comments
W3: What the results show
We would like to clarify that we do not assume the pretrained predictor is close to the target solution. Our assumptions concern its support and signs, not the magnitude of the coefficients. The pretrained predictor can be arbitrarily far from the target in distance.
In the limiting saddle-to-saddle dynamic, the magnitudes of the pretrained coefficients do not affect the trajectory. Only their support and signs matter for each saddle. The main result is therefore not simply that starting close to the solution improves learning. We characterize how fine-tuning uses the information contained in the initial support, in a non-convex setting. More precisely, we show how correctly inherited coordinates can reduce the sample complexity, how initially incorrect signs can be corrected, and why inherited false positives may remain in the final support. Our results also quantify the effect of initialization quality through the number of coordinates that must be learned and the number of inherited false positives. Figure 6 illustrates precisely why Euclidean proximity is not sufficient: on the same five datasets, an initialization containing six of the eight true coordinates with correct signs and magnitude 5 has a much larger initial distance but achieves lower final test MSE than some closer initializations with incorrect signs or false positives, despite all runs reaching a training loss below .
This focus on support overlap rather than proximity is motivated by the non-convex nature of the underlying optimization problem. Distance-based guarantees are common in fine-tuning theory and we agree with the reviewer that they offer limited insight. Our goal is precisely to go beyond this criterion.
Response to Presentation Comments
We thank the reviewer for these suggestions. We agree that some definitions and explanations should be made clearer.
First, support refers to the standard definition, given by the set of indices where a vector has nonzero coefficients: . We will add this definition explicitly.
Second, the term contaminated face refers to a set s.t. and , meaning that the current saddle contains coordinates in . We agree that this terminology is not very clear and will replace it with a more explicit definition.
For the first sentence example, we will change it to the simpler phrasing:
This suggests a statistical gain: the factor should only be paid for coordinates whose membership in the target support is unknown.
Regarding the second cited sentence, we are not sure what the reviewer finds unclear, but we would be happy to provide further clarification. Broadly speaking, this sentence describes in words how our idealized stopping rule is defined, while also explaining its purpose: although this stopping rule is not directly implementable in practice, it allows us to establish strong theoretical guarantees.
Thank you for your detailed rebuttal, which I have read carefully. While I appreciate the clarification of several points, there are some issues which motivate me to retain my score.
It seems that many choices in the analysis are motivated by known aspects of diagonal linear networks (DLN), such as the implicit bias or sparsity of solutions. However, I do not see much evidence suggesting that DLNs themselves are an accurate representation of neural network training. Even non-diagonal deep linear networks would be a better setting to study, although they also do not reflect practical neural networks.
I agree that only restricting the support and signs is indeed a weaker requirement than having the coefficients be physically close to optimal. However, even assuming that the pretrained model can have some of the same supports and signs seems like a strong assumption presented as given. This is especially since pretraining is usually a semi-supervised/self-supervised task, not a regression task. Modeling this process (empirically or theoretically) as a model first trained on one regression task and then fine-tuned on another regression task does not really capture the main considerations of the pretraining/finetuning process, as mentioned in my original review.
Finally, I believe the manuscript continues to suffer from clarity issues beyond the examples I pointed out, which may not be easily fixed in the meantime. Due to this and because I believe the theoretical setting has limited relevance to the real considerations of finetuning, I will retain my score.
We thank the reviewer for their follow-up reply, which we read with slight perplexity given the rather dense and well-acknowledged literature on neural network theory that focuses on diagonal linear networks. Indeed, this architecture has been recognized by the community as the simplest proxy and a necessary first step toward understanding neural network training.
Finally, we politely reiterate that the implicit regularization of gradient flow for linear networks is either not precisely known (for matrix completion/sensing) or biased toward the -norm (for scalar-output regression). The latter setting, which has been previously studied (see, e.g., Shachaf et al., 2021) assuming perfect pretraining, allows for an -perturbation-based analysis. We emphasize that our analysis goes beyond such adaptation and rigorously describes the learning of genuinely new features.
Therefore, studying linear networks in our setting is either theoretically out of reach or not interesting with respect to existing work.
We thank the reviewer for their valuable feedback and will carefully address all the remarks in the revised version.
W1: Choice of the theoretical setting
Our theoretical setting is indeed deliberately simple. We made this choice precisely to enable a rigorous and precise analysis of the fine-tuning dynamics. DLNs are commonly used as tractable models that allow for complete theoretical analyses, not only of their limit points, but also, for instance, of their full training dynamics in the vanishing-initialization regime, while retaining the non-convexity of the training loss and key dynamical behaviors observed in neural network training. In this sense, we view DLNs as a natural model for obtaining a tractable yet non-trivial analysis of fine-tuning dynamics. By contrast, a complete characterization of the training dynamics of deep linear networks remains beyond the reach of current theoretical techniques.
The choice of sparse linear regression, rather than general linear regression, is also natural in our setting, since DLNs are known to favor sparse estimators. More importantly, previous theoretical works on fine-tuning have typically considered linear regression settings in which the fine-tuning parameters remain close in ℓ2-norm to the pre-trained ones. We believe that such a setting does not fully capture a central aspect of fine-tuning: the ability to reuse features acquired during pre-training while simultaneously learning genuinely new ones. This goes beyond viewing fine-tuning as merely a small ℓ2-norm perturbation of the pre-trained weights. In contrast, sparse regression provides a simple setting in which this feature-reuse/feature-learning structure can be explicitly modeled and theoretically analyzed.
W2: Pretraining and distribution shift
Our goal is to study how a pre-trained initialization can improve learning on a downstream task. We therefore treat pre-training as a black box and assume that the initial predictor β0 is fixed, possibly with incorrectly inherited support and/or signs.
Nevertheless, we view this initialization as an idealization of what would be obtained by pre-training a DLN, from a small-scale initialization, on a previous and potentially different regression task. This interpretation is motivated by existing results on the implicit bias of DLNs, which show that, as the balancedness parameter μ at initialization tends to zero (with an uninformed initialization), the final predictor converges to a minimum-ℓ1-norm interpolator, and is therefore naturally sparse. Our initialization scheme can thus be viewed as modeling the outcome of such a pre-training procedure in the vanishing-initialization limit.
From this perspective, the difference between the inherited from pre-training parameter β0 and the downstream ground-truth parameter β⋆ quantifies the distribution shift between the pre-training and fine-tuning tasks. For instance, one may think of β0 as the ground-truth parameter of the pre-training task, with the support overlap quantifying how much of the relevant structure is shared between the two tasks. Figure 10 further illustrates this setting experimentally, by pre-training a DLN on a source regression task and directly fine-tuning its learned factors on a different target task, without thresholding. Fine-tuning improves test MSE compared to training from scratch.
Finally, our work focuses on supervised fine-tuning. We view parameter-efficient fine-tuning as an algorithmic approach motivated by different considerations and therefore outside the scope of this work. Regarding the question of "large non-convex models with suboptimal stationary points", DLNs already exhibit precisely these features (see answer to W1 above). They thus provide a suitable model that retains such non-trivial optimization phenomena while remaining sufficiently tractable to allow for a precise theoretical analysis of the fine-tuning dynamics.