acceptodds
Under review as a conference paper at ICLR 2027

Statistical Benefits of Fine-Tuning from Pretrained Initialization in Diagonal Linear Networks

Abstract

Adapting pretrained models to downstream tasks with limited data has become a central paradigm in modern deep learning. Yet, despite its widespread practical success, how fine-tuning leverages information from pretraining remains poorly understood theoretically. We study fine-tuning from pretrained weights through the lens of sparse linear regression and two-layer diagonal linear networks. In our setting, pretraining provides information through the support (and signs) of the initialization predictor, which may contain coordinates relevant to the downstream task. We show how pretrained information reshapes the implicit bias and training dynamics, and can thereby reduce the sample complexity of recovering the target parameters and support. In particular, for a clean initialization with correctly inherited signs, we show that the required sample size is comparable to that of a weighted Lasso estimator that explicitly exploits the pretrained support through a suitably chosen regularizer. Our results thus show how information encoded in pretrained weights can be implicitly exploited by gradient-based fine-tuning, reducing the amount of data needed to recover a downstream task.

Primary area:
learning theory
Keywords:
diagonal linear networks, saddle-to-saddle dynamics, implicit bias, fine-tuning, sparse recovery, early stopping

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HNGQOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper addresses the fine-tuning paradigm, where a large pretrained foundation model is trained further on some downstream dataset. Specifically, the work aims to characterize how the pretrained initialization benefits the fine-tuning process. The authors consider sparse linear regression and two-layer diagonal linear neural networks. They consider a setup where the model is initialized such that it already has partial information about the target vector, for example if some of the parameters already correctly match the target parameters. They demonstrate that gradient descent will exploit these similarities to converge with improved sample complexity.

Critical strengths

The overall problem is interesting and important.

Critical weaknesses
  1. The settings examined in the paper are highly constrained and not well motivated. Why consider sparse linear regression, and not general linear regression? Why consider diagonal linear neural networks, and not general linear neural networks? These limit the usefulness of the theoretical results.
  2. The analysis does not really capture core issues of the fine-tuning framework. First, it does not model the pretraining process, taking the pretrained model as given. It assumes the model is initialized such that it already correctly identifies relevant coordinates, i.e. correctly models some of the data, which is a strong assumption. A core concern of fine-tuning is the distribution shift between the pretraining data and fine-tuning data, which this analysis does not capture. The analysis also does not consider other concerns of fine-tuning, such as parameter-efficient fine-tuning, self-supervised pretraining vs. supervised fine-tuning, or large nonconvex models with suboptimal stationary points.
  3. Ultimately, the conclusion of this work may be summarized as, if the model is initialized at a point where it is close to the desired solution, then it will converge with fewer samples to the desired solution. This seems somewhat circular and does not reveal new insights.

Additional weakness

  1. The use of the word "support" is somewhat confusing. The paper would benefit from defining it in one line. I understand it is to mean the indices where the vector is non-zero (as opposed to a support vector).
  2. Some terminology seems unclear and nonstandard. For example, what is a "contaminated face"?
  3. The writing is dense and convoluted, making it difficult to discern the main contribution of this work. An example of this is the following sentence:

"This suggests a statistical gain: the ambient dimensional logarithmic cost should only be incurred for coordinates whose membership in the target support is unknown, while the cost of estimating the coefficients on all s coordinates remains."

Another example is:

"We first consider an ideal stopping rule which stops the first time, if finite, S⋆S^⋆ is included in the estimated support; and returns the corresponding saddle. This rule is not available in practice, but isolates the main statistical question: does the trajectory recover the remaining signal before proposing a new null coordinate?"

This work would benefit from rewriting with the goal of making it understandable to a human reader.

Questions for authors

See questions from the above section.

4 comments

We thank the reviewer for their valuable feedback and will carefully address all the remarks in the revised version.

W1: Choice of the theoretical setting

Our theoretical setting is indeed deliberately simple. We made this choice precisely to enable a rigorous and precise analysis of the fine-tuning dynamics. DLNs are commonly used as tractable models that allow for complete theoretical analyses, not only of their limit points, but also, for instance, of their full training dynamics in the vanishing-initialization regime, while retaining the non-convexity of the training loss and key dynamical behaviors observed in neural network training. In this sense, we view DLNs as a natural model for obtaining a tractable yet non-trivial analysis of fine-tuning dynamics. By contrast, a complete characterization of the training dynamics of deep linear networks remains beyond the reach of current theoretical techniques.

The choice of sparse linear regression, rather than general linear regression, is also natural in our setting, since DLNs are known to favor sparse estimators. More importantly, previous theoretical works on fine-tuning have typically considered linear regression settings in which the fine-tuning parameters remain close in ℓ2\ell_2-norm to the pre-trained ones. We believe that such a setting does not fully capture a central aspect of fine-tuning: the ability to reuse features acquired during pre-training while simultaneously learning genuinely new ones. This goes beyond viewing fine-tuning as merely a small ℓ2\ell_2-norm perturbation of the pre-trained weights. In contrast, sparse regression provides a simple setting in which this feature-reuse/feature-learning structure can be explicitly modeled and theoretically analyzed.

W2: Pretraining and distribution shift

Our goal is to study how a pre-trained initialization can improve learning on a downstream task. We therefore treat pre-training as a black box and assume that the initial predictor β0\beta_0 is fixed, possibly with incorrectly inherited support and/or signs.

Nevertheless, we view this initialization as an idealization of what would be obtained by pre-training a DLN, from a small-scale initialization, on a previous and potentially different regression task. This interpretation is motivated by existing results on the implicit bias of DLNs, which show that, as the balancedness parameter μ\mu at initialization tends to zero (with an uninformed initialization), the final predictor converges to a minimum-ℓ1\ell_1-norm interpolator, and is therefore naturally sparse. Our initialization scheme can thus be viewed as modeling the outcome of such a pre-training procedure in the vanishing-initialization limit.

From this perspective, the difference between the inherited from pre-training parameter β0\beta_0 and the downstream ground-truth parameter β⋆\beta^\star quantifies the distribution shift between the pre-training and fine-tuning tasks. For instance, one may think of β0\beta_0 as the ground-truth parameter of the pre-training task, with the support overlap quantifying how much of the relevant structure is shared between the two tasks. Figure 10 further illustrates this setting experimentally, by pre-training a DLN on a source regression task and directly fine-tuning its learned factors on a different target task, without thresholding. Fine-tuning improves test MSE compared to training from scratch.

Finally, our work focuses on supervised fine-tuning. We view parameter-efficient fine-tuning as an algorithmic approach motivated by different considerations and therefore outside the scope of this work. Regarding the question of "large non-convex models with suboptimal stationary points", DLNs already exhibit precisely these features (see answer to W1 above). They thus provide a suitable model that retains such non-trivial optimization phenomena while remaining sufficiently tractable to allow for a precise theoretical analysis of the fine-tuning dynamics.

W3: What the results show

We would like to clarify that we do not assume the pretrained predictor is close to the target solution. Our assumptions concern its support and signs, not the magnitude of the coefficients. The pretrained predictor can be arbitrarily far from the target in ℓ2\ell_2 distance.

In the limiting saddle-to-saddle dynamic, the magnitudes of the pretrained coefficients do not affect the trajectory. Only their support and signs matter for each saddle. The main result is therefore not simply that starting close to the solution improves learning. We characterize how fine-tuning uses the information contained in the initial support, in a non-convex setting. More precisely, we show how correctly inherited coordinates can reduce the sample complexity, how initially incorrect signs can be corrected, and why inherited false positives may remain in the final support. Our results also quantify the effect of initialization quality through the number of coordinates that must be learned and the number of inherited false positives. Figure 6 illustrates precisely why Euclidean proximity is not sufficient: on the same five datasets, an initialization containing six of the eight true coordinates with correct signs and magnitude 5 has a much larger initial distance but achieves lower final test MSE than some closer initializations with incorrect signs or false positives, despite all runs reaching a training loss below 10−810^{-8}.

This focus on support overlap rather than ℓ2\ell_2 proximity is motivated by the non-convex nature of the underlying optimization problem. Distance-based guarantees are common in fine-tuning theory and we agree with the reviewer that they offer limited insight. Our goal is precisely to go beyond this criterion.

Response to Presentation Comments

We thank the reviewer for these suggestions. We agree that some definitions and explanations should be made clearer.

First, support refers to the standard definition, given by the set of indices where a vector has nonzero coefficients: supp(β)={i:βi≠0}\text{supp}(\beta)=\{i:\beta_i\neq0\}. We will add this definition explicitly.

Second, the term contaminated face refers to a set E=A∪BE = A \cup B s.t. A⊂S⋆A \subset S^{\star} and B⊂F0B \subset F_{0}, meaning that the current saddle contains coordinates in S⋆ cS^{\star\,c}. We agree that this terminology is not very clear and will replace it with a more explicit definition.

For the first sentence example, we will change it to the simpler phrasing:

This suggests a statistical gain: the log⁡(d/s)\log(d/s) factor should only be paid for coordinates whose membership in the target support is unknown.

Regarding the second cited sentence, we are not sure what the reviewer finds unclear, but we would be happy to provide further clarification. Broadly speaking, this sentence describes in words how our idealized stopping rule is defined, while also explaining its purpose: although this stopping rule is not directly implementable in practice, it allows us to establish strong theoretical guarantees.

Thank you for your detailed rebuttal, which I have read carefully. While I appreciate the clarification of several points, there are some issues which motivate me to retain my score.

It seems that many choices in the analysis are motivated by known aspects of diagonal linear networks (DLN), such as the implicit bias or sparsity of solutions. However, I do not see much evidence suggesting that DLNs themselves are an accurate representation of neural network training. Even non-diagonal deep linear networks would be a better setting to study, although they also do not reflect practical neural networks.

I agree that only restricting the support and signs is indeed a weaker requirement than having the coefficients be physically close to optimal. However, even assuming that the pretrained model can have some of the same supports and signs seems like a strong assumption presented as given. This is especially since pretraining is usually a semi-supervised/self-supervised task, not a regression task. Modeling this process (empirically or theoretically) as a model first trained on one regression task and then fine-tuned on another regression task does not really capture the main considerations of the pretraining/finetuning process, as mentioned in my original review.

Finally, I believe the manuscript continues to suffer from clarity issues beyond the examples I pointed out, which may not be easily fixed in the meantime. Due to this and because I believe the theoretical setting has limited relevance to the real considerations of finetuning, I will retain my score.

We thank the reviewer for their follow-up reply, which we read with slight perplexity given the rather dense and well-acknowledged literature on neural network theory that focuses on diagonal linear networks. Indeed, this architecture has been recognized by the community as the simplest proxy and a necessary first step toward understanding neural network training.

Finally, we politely reiterate that the implicit regularization of gradient flow for linear networks is either not precisely known (for matrix completion/sensing) or biased toward the ℓ2\ell_2-norm (for scalar-output regression). The latter setting, which has been previously studied (see, e.g., Shachaf et al., 2021) assuming perfect pretraining, allows for an ℓ2\ell_2-perturbation-based analysis. We emphasize that our analysis goes beyond such adaptation and rigorously describes the learning of genuinely new features.

Therefore, studying linear networks in our setting is either theoretically out of reach or not interesting with respect to existing work.

Reviewer_TdhoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies the statistical benefits of fine-tuning diagonal linear networks for sparse linear regression, starting from a pretrained initialization that contains partial information about the target support. Using a mirror-flow characterization, the authors show that the gradient-flow solution in the noiseless setting is characterized by an implicit regularizer that favors coordinates and signs inherited from pretraining. They establish sample-complexity guarantees for exact recovery that can improve over standard sparse regression methods when the initialization is sufficiently informative. They further analyze the noisy setting through the limiting saddle-to-saddle dynamics and show that an appropriately early-stopped trajectory recovers the true support, potentially retaining only inherited false positives, with favorable estimation-error guarantees.

Critical strengths

I think this is an interesting and well-motivated theoretical work addressing an important question about the statistical benefits of pretrained initialization. The connection between gradient flow, implicit regularization, and weighted sparse recovery is insightful. I also appreciate that the authors go beyond characterizing the convergence point and analyze the noisy setting through early-stopping dynamics, obtaining meaningful support-recovery and estimation guarantees.

Critical weaknesses

My main concern is that the paper assumes a pretrained initialization with useful support and sign information without modeling how such information is obtained. It would be interesting to justify this assumption, for example by considering pretraining on a mixture of regression tasks and studying whether the learned signs transfer to a downstream task.

Some parts of the presentation could also be improved. The abstract should clarify the assumptions on pretraining, and Section 3.2 would benefit from more background on mirror flow. The distinction between correcting initially incorrect signs and potentially retaining inherited false positives could also be explained more clearly.

Questions for authors

Are the results extendable to subgaussian input instead of gaussian?

Is the characterization of the convergence point of gradient flow with the mirror flow analysis subject to the convergence of the gradient flow in the first place? Or can you also show that the gradient flow converge in bounded time in the first place?

1 comment

We thank the reviewer for their valuable feedback and will carefully address all the remarks in the revised version.

W1: Pretraining assumptions

Our goal is to study how a pretrained initialization can improve learning on a downstream task. We treat pretraining as a black box and assume that the initial predictor β0\beta^0 is fixed and might have incorrect inherited support and/or signs.

However, we view this initialization as an idealization of what would be obtained after pre-training a DLN (with small scale initialization) on a previous (potentially different) regression task. Such an idealization is motivated by the existing works on the implicit bias of DLNs, which show that, as the balancedness parameter μ\mu goes to 0 at initialization (with β0=0\beta_0=0, ie uninformed init) the final predictor corresponds to a minimal ℓ1\ell_1-norm interpolator, i.e, a sparse parameter. Our initialization scheme thus models what would be obtained from such a pre-training in the limit of a vanishing pre-training initialization. In practice, the parameter obtained after pre-training is not exactly sparse, but only approximately in the sense that many coordinates are close to 0. We have added experiments (see Figure 10) in the revised version (available on OpenReview), where we first pre-train a DLN on a regression task, and then fine-tune the obtained weights on a different, fine-tuning regression task. The source and target supports both have size 100 and share exactly 90 coordinates, with identical signs on the overlapping coordinates. The learned DLN factors are directly reused for fine-tuning, without any thresholding. Although the initialization before fine-tuning is only approximately sparse in this setting, the empirical observations are consistent with our theory. This supports that our choice of initialization is a faithful idealization of real pre-training while enabling a clean theoretical analysis.

W2: Presentation

We apologize if some parts of the paper were unclear to the reviewer. In the revised version, we will explain more clearly in both the abstract and main paper that our initialization can be seen as an idealization from a previous pre-training on a regression task. We will also add in the appendix a small section recalling the connection between gradient flow on DLNs parameters, and mirror flow on the effective parameters β\beta, and what are the classical properties of mirror flow that are useful to us (besides the new Figure 2 illustrating the convergence to the Saddle to saddle process as μ→0\mu\to0). Also, we will make it clearer when introducing the setting what we mean by the different terms as incorrect signs, inherited false positives, etc. The new Figure 7 also illustrates empirically in more details how these different terms affect the support recovery of the ground truth β⋆\beta^\star.

Q1: Sub-Gaussian designs

We believe that an extension to sub-Gaussian designs may be possible, although it is not immediate, as some parts of our proofs rely on properties specific to Gaussian random variables. In particular, we use chi-square tail bounds to control gradients and least-squares coefficients. For designs with independent sub-Gaussian entries, some of these arguments could be replaced by concentration inequalities such as Hanson-Wright. Establishing such an extension would nevertheless require additional technical work, while we expect it to provide limited additional insight and leave our main findings essentially unchanged.

Q2: Convergence of gradient flow

Convergence of gradient flow to a limit point is not an assumption but follows from its mirror-flow formulation. In Appendix A.3 of the revised version we establish a stronger result, proving convergence as t→∞t\to\infty through uniform-in-μ\mu bounds, which also allow us to control the trajectories as μ→0\mu\to0 under the given time rescaling.

Reviewer_uewPOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies the statistical benefits of fine-tuning from a nonzero pretrained initialization in two-layer diagonal linear networks. In the vanishing-layer-imbalance limit, it characterizes a leading implicit regularizer that assigns an ℓ1\ell_1 penalty to previously unused coordinates, no penalty to inherited coordinates that preserve their signs, and twice the ℓ1\ell_1 penalty when their signs are reversed. It then establishes initialization-dependent sample-complexity guarantees for noiseless recovery and analyzes support recovery and early stopping along the limiting saddle-to-saddle (S2S) trajectory under noisy observations.

Critical strengths
  1. I find the connection between inherited signed support and statistical efficiency insightful. The implicit-regularizer characterization makes the effects of reusing coordinates and reversing inherited signs explicit. The recovery guarantees also distinguish the cost of estimating coefficients from that of identifying coordinates that remain to be learned, providing a clear account of how informative initialization can reduce downstream sample requirements.

  2. The paper analyzes the order in which coordinates become active along the S2S trajectory from a pretrained initialization. Under suitable sample-size and signal-strength conditions, it shows that all remaining true coordinates are recovered before any new false positives are introduced, while accounting for inherited sign errors and false positives. Together with the proposed stopping rule, this gives a statistical justification for early stopping in the noisy setting.

Critical weaknesses
  1. The justification of Theorem 4 in Appendix A.3 appears incomplete as written. The proof invokes the uniform path-length bound and compactness arguments of Pesme and Flammarion (2023), but the present setting retains a fixed nonzero pretrained predictor, whereas the cited analysis starts from a zero predictor. In particular, the nonzero initial mirror-map term must be accounted for when establishing bounds uniform in the layer imbalance μ\mu. The manuscript does not explicitly establish the corresponding uniform boundedness of the iterates and path length in this setting. Assumption 3 addresses uniqueness of the constrained saddles, but does not by itself establish these bounds. A more explicit extension of the relevant arguments is needed, since this convergence result is the bridge between the S2S recovery guarantees and their interpretation as guarantees for gradient-flow fine-tuning.
  2. The experiments in Appendix F directly simulate the limiting S2S algorithm, rather than training the diagonal linear network with finite layer imbalance. They therefore illustrate the statistical behavior of the limiting process, but do not test how well it describes finite-imbalance fine-tuning. A comparison with numerical gradient flow or small-step gradient descent, across several imbalance scales and with the prescribed time rescaling, would strengthen this connection. Moreover, the experiments do not systematically vary inherited sign errors or false positives, despite these being central to the theoretical results. Evaluating coefficient trajectories, prediction error, and variable selection with an explicitly specified threshold would help establish the empirical relevance of these aspects of the theory.
Questions for authors
  1. The guarantees rely on the exact support of the pretrained predictor, whereas standard gradient-based pretraining does not generally produce exact zeros at finite time. With the pretrained predictor fixed as μ→0\mu \to 0, even arbitrarily small nonzero coefficients are treated as inherited coordinates. For a dense initialization, this gives s+f0=ds+f_0=d, substantially weakening the sample-efficiency guarantee. Can the authors clarify whether comparable benefits hold for approximately sparse initializations, and how the analysis would depend on the magnitude of small coefficients relative to μ\mu, or on a thresholding step before fine-tuning?
  2. The null-gradient stopping rule requires the unknown quantities mm and f0f_0, or a known upper bound B≥m+f0B \ge m+f_0. Although Appendix D.5 provides the observable choice B=dB=d, this introduces an mdmd term into the sufficient sample-size requirement and can substantially weaken the advertised sample-efficiency benefit. How can an informative bound BB be obtained without access to the target support and signs? Could the authors discuss a data-driven calibration and evaluate sensitivity to overestimating or underestimating m+f0m+f_0, including the resulting trade-off between premature stopping and false-positive inclusion?

Overall assessment

I lean toward rejection in the current form. The paper offers an insightful characterization of how pretrained signed support affects implicit bias and recovery along the S2S trajectory. However, my main concern is whether the extension from zero to nonzero initialization in Theorem 4 is sufficiently justified. This result is central to interpreting the S2S recovery guarantees as statements about gradient-flow fine-tuning, and the current proof does not make the required uniform bounds explicit in the new setting. The experiments also remain confined to the limiting S2S algorithm, leaving the finite-imbalance behavior untested. Clarifying the convergence argument would be the most important factor in reconsidering my assessment; finite-imbalance experiments and a more practical calibration of the stopping rule would further strengthen the paper.

3 comments

We thank the reviewer for their valuable feedback and will carefully address all the remarks in the revised version.

Our main contributions are to characterize how pretraining information encoded in the initial support and signs changes the implicit bias and to establish support-recovery guarantees depending on this information. Importantly, we do not assume that the pretrained predictor is close to the target in ℓ2\ell_2 distance. We show how correctly inherited coordinates can reduce the sample complexity, how initially incorrect signs can be corrected, and why inherited false positives may remain in the final support.

W1: Convergence to Saddle-to-saddle trajectory

We added in Appendix A.2 and A.3 of the revised version (available on openreview) a self contained, rigorous proof of Theorem 4. From a high level, it first shows that the predictor trajectories are uniformly bounded in both μ\mu and time. From there, the proof follows similar lines to the one of Pesme and Flammarion (2023).

W2: Experiments

We agree that the initial experiments mainly studied the limiting S2S algorithm and did not directly evaluate finite-imbalance gradient flow. To address this issue, we have added three additional experiments (Figures 2, 7 and 10 in the Appendix). In Figure 2, we compare the saddle-to-saddle algorithm with the gradient flow and gradient descent of the DLN for several values of μ\mu, using the time rescaling λμ=12log⁡(1/μ)\lambda_\mu=\frac12\log(1/\mu). Figure 2 shows increasingly sharp transitions approaching the S2S plateaus as μ\mu decreases, supporting the convergence of the gradient flow to the S2S algorithm as μ→0\mu\to0.

Figure 7 studies the effect of initialization errors more systematically. We vary the number of inherited wrong signs w0w_0 and false positives f0f_0, and measure signed-support recovery using the signed-F1 score. Finally, Figure 10 considers actual DLN pre-training on a different regression task, followed by fine-tuning with gradient descent. The source and target supports both have size 100 and share exactly 90 coordinates, with identical signs on the overlapping coordinates. The pre-trained DLN factors are directly reused for fine-tuning, without thresholding. The results show that fine-tuning improves test MSE compared to training from scratch and confirm the theoretical findings obtained with our idealized initialization scheme (see below).

Q1: Approximately sparse initialization

Our analysis assumes that the pretrained predictor β0\beta^0 is fixed as μ→0\mu\to0. In this setting, the limiting saddle-to-saddle dynamic depends only on its support and signs, not on the magnitudes of its nonzero coefficients.

We agree that a dense initialization may remove the statistical benefit in our bounds. Our assumption of exact zeros in β0\beta^0 idealizes pre-training a DLN in the vanishing-μ\mu regime, whose implicit bias favors sparse solutions. In practice, coefficients are only approximately zero. As illustrated in Figure 10, genuine DLN pre-training followed by fine-tuning yields observations consistent with our theoretical analysis. This does not require knowledge of the target support (S⋆)(S^\star) nor any thresholding scheme. Our current guarantees concern a fixed initialization as μ→0\mu\to0, rather than a joint limit in which the pretrained coefficients also vanish. We believe that our analysis could be extended to non-zero but small coordinates, provided that they vanish at a rate proportional to μ\mu. We leave a rigorous treatment of this extension for future work.

Q2: Stopping-rule calibration

As raised in the paper, the current stopping rule indeed requires the knowledge of an upper bound BB of m+f0m+f_0. We think this is a rather minor weakness, as the main goal of this section is to mainly show that one can infer an optimal stopping time, and in practice, such a stopping time would be decided by tracking some validation loss.

However, we recently figured out that it is indeed possible to propose a calibration that does not require any prior bound on m+f0m+f_0. The idea is to separate the candidate faces according to their Hamming distance ∣A△Sinit∣|A\triangle S_{\mathrm{init}}| from the initial support, and allocate a confidence budget ηk\eta_k to each face of Hamming distance kk. One can choose ηk=η(k+1)(k+2)(dk)\eta_k = \frac{\eta}{(k+1)(k+2)\binom{d}{k}}.

Using this calibration rule should yield a sample complexity requirement of order s+f0+m(m+f0)log⁡(d/(m+f0))+mlog⁡(d/η)s + f_0 + m(m + f_0) \log(d/(m + f_0)) + m \log(d/\eta), which is relatively similar to the s+f0+mB+mlog⁡(d/η)s+f_0+mB+m\log(d/\eta) requirement with the current rule. We plan on keeping the current BB-based formulation for the sake of simplicity, but we will mention the possibility of this calibration rule in the revised version. If the reviewer believes that a rigorous proof of this extension would strengthen the paper, we would be happy to include it in the revised version.

Thank you for the detailed response and revisions. The expanded argument in Appendix A.3 substantially addresses my main concern: it now explicitly treats the nonzero initialization when establishing uniform boundedness of the predictors, and then derives the path-length bound needed for the convergence analysis. The added gradient-flow/GD comparisons and initialization-error experiments also address the main scope limitations of the original experiments.

The dependence on exact sparsity and on prior information for stopping-rule calibration remains a limitation. In particular, the direct pretraining experiment provides useful empirical support but does not extend the guarantees to approximately sparse initializations. I also did not find the proposed Hamming-distance-based calibration established in the attached revision, so I regard it as a possible extension rather than a proved result.

Overall, the revisions substantially strengthen the paper and address my principal reasons for the initial negative assessment. I am therefore raising my score from 2 to 3.

We thank the reviewer for their reply and for increasing their score.

In Figure 11 of the revised version, we have added an experiment showing that (a) the pre-trained weights already used in Figure 10 are indeed approximately sparse, and (b) fine-tuning from the pruned weights slightly improves the final loss but does not appear to significantly change the dynamics.

We also provide a proof sketch of the Hamming-distance-based calibration below. A more detailed proof can be added to the final version if the reviewer wants.

Thank you again,

The authors


Proof sketch:

The idea is to distribute the confidence budget over candidate oracle faces according to their Hamming distance k=∣A△Sinit∣k=|A\triangle S_{\mathrm{init}}| from the initial support.

There are at most (dk)\binom{d}{k} such faces at distance kk, with signs fixed as in the recovery analysis. Assigning to each face the confidence budget ηk=η(k+1)(k+2)(dk)\eta_k=\frac{\eta}{(k+1)(k+2)\binom{d}{k}}, we can apply the Gaussian concentration bound of Lemma 5.1, as in Lemma 6.1. A union bound over all faces gives $\mathbb{P}(\text{at least one bound fails}) \leq \sum_{k=0}^{d}\binom{d}{k}\eta_k\leq\eta$. Thus, the null-gradient bound holds simultaneously on all candidate oracle faces, without requiring any prior knowledge of m+f0m+f_0. The same first-failure argument as in the proof of Theorem 2.2 then ensures that no new null coordinate is accepted before stopping. The stopping threshold only uses the observed distance k=∣A△Sinit∣k=|A\triangle S_{\mathrm{init}}|.

Under the same assumptions, the existing detectability argument yields a sample complexity of order s+f0+m(m+f0)log⁡edm+f0+mlog⁡dηs+f_0+m(m+f_0)\log\frac{ed}{m+f_0} +m\log\frac{d}{\eta}, which is comparable to the current s+f0+mB+mlog⁡(d/η)s+f_0+mB+m\log(d/\eta) requirement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.