acceptodds
Under review as a conference paper at ICLR 2027

Be loud and relevant: pretraining of compositional tasks in linear RNNs

Abstract

Pretraining models with a curriculum of simpler tasks is a common approach to speed up training, but it is unclear what aspects of common task structure benefit learning, or how to choose a theoretically well-justified curriculum. Here we leverage recent learning theory in linear RNNs (Proca et al., 2025), and derive criteria for useful curricula that extend to compositional tasks. We find that pretraining tasks speed up learning and lead to more stereotyped solutions when they train model weights to large magnitude, while also building related structure to the target task. This mirrors results in feedforward networks, but does not rely on previously used statistical arguments of infinite-width networks. Instead we provide a mechanistic description for what constitutes a good CL task in RNNs: we demonstrate that factorized tasks representing the constituent parts of a composed task are ideal curricula because they possess hyperbolic learning dynamics in their pretraining phase that bypass saddles and slow points of the loss landscape.

Primary area:
applications to neuroscience & cognitive science
Keywords:
curriculum learning, learning theory, pretraining, linear RNNs

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_td1MOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work studies when simpler-task pretraining accelerates target-task training in linear RNNs. Extending existing learning-dynamics analysis to temporally structured task covariances with fixed recurrent weights, the authors derive speedup conditions for factorized, geometrically aligned tasks under proper initialization, governed by covariance strength and pretraining accuracy.

Evaluating multiple curricula on a 2D compositional task, they find sufficiently scaled factorized pretraining reduces training time and solution variance. The results are explained via the saddle avoidance, with supplementary experiments covering nonlinear activations, mismatched task geometry, and trainable recurrence.

Critical strengths
  1. the paper analytical treatment of total training time.

Equations 9–10 include the costs of pretraining when comparing a curriculum with direct target training which makes the analysis relevant to actual optimization savings. Although its conclusions depend on restrictive assumptions about task geometry and initialization.

  1. Controlled comparisons of curriculum structure.

The compositional experiments evaluate factorized tasks, scaled targets, target permutations, low-rank tasks and random tasks at varying amplitudes. These controls suggest the observed speedup cannot be attributed to weight magnitude alone.

Critical weaknesses
  1. The central geometric explanation needs stronger verification

In the paper, section 4.2 describes the zero-weight solution as a “local maximum”. However, for a nontrivial factorized mode, however, the loss near the origin contains a term proportional to �0��0� which aligned and oppositely signed perturbations respectively decrease and increase the loss and made the origin a saddle in the unrestricted parameter space.

Moreover, loss plateaus, gradient norms, and a two-dimensional PCA slice do not establish that compositional trajectories avoid saddles in the full parameter space. Since saddle avoidance is the primary mechanistic explanation for the reported behavior, these issues affect the interpretation itself, not merely the terminology.

  1. The compositional evidence remains narrow

The compositional study is restricted to two hidden units and two input/output channels (Appendix A.4). Although the supplementary ReLU and jointly trained recurrence experiments extend portions of the covariance analysis, they do not demonstrate that the compositional mechanism generalizes to higher-dimensional tasks. In addition, the joint-training experiment initializes from parameters already constructed near an optimum, which limits its ability to probe learning dynamics from general starting conditions. Consequently, the current evidence does fully support authors assumptions.

Questions for authors

1, See Critical Weakness 1

2, See Critical Weakness 2

3, The paper does not discuss [1]'s analysis of how task alignment and initialization affect transfer in deep linear networks. Could author compare the predicted transfer conditions in the identity recurrence limit and clarify which effects specifically require temporal recurrence?

4, The paper states that the rescaling symmetry guarantees a2=c2a^2=c^2 (Appendix A.1). Gradient flow generally conserves a2−c2a^2-c^2, while equality requires balanced initialization. Can author state this assumption explicitly and explain how the training-time criterion changes for unbalanced initialization?

[1] Andrew K. Lampinen and Surya Ganguli. “An Analytic Theory of Generalization Dynamics and Transfer Learning in Deep Linear Networks.” ICLR, 2019.

4 comments
AuthorsOpenReview ↗

Reply to reviewer comments 1/2

We have addressed the reviewer concerns below. To ease discussion, we have reiterated and numbered each concern.

1a. "The central geometric explanation needs stronger verification. In the paper, section 4.2 describes the zero-weight solution as a “local maximum”. However, for a nontrivial factorized mode, however, the loss near the origin contains a term proportional to which aligned and oppositely signed perturbations respectively decrease and increase the loss and made the origin a saddle in the unrestricted parameter space. "

Reply: The reviewer is correct that the zero-weight solution is not a local optimum. We numerically calculated the hessian of this solution and it is indeed a saddle. We had not fully appreciated the structure of the null solution, and thank the reviewer for the careful reading of our derivation. We have updated it in the main text.

1b. "Moreover, loss plateaus, gradient norms, and a two-dimensional PCA slice do not establish that compositional trajectories avoid saddles in the full parameter space. Since saddle avoidance is the primary mechanistic explanation for the reported behavior, these issues affect the interpretation itself, not merely the terminology."

Reply: The optimizations do not perfectly traverse through saddles, this is true. The norm of the gradient never fully diminishes. We believe this is because introducing compositional structure altered the nature of the fixed point. We had previously reported the norm of the gradients in the 2D PCA space in Supp. Fig. 8, where you can see clear saddle structure in the factorized setting that is smoothed out in the compositional case, suggesting a change in the fixed points’ characterization to a slow point. We have changed the language in the main text to reflect this point.

2a. "The compositional evidence remains narrow. The compositional study is restricted to two hidden units and two input/output channels (Appendix A.4). Although the supplementary ReLU and jointly trained recurrence experiments extend portions of the covariance analysis, they do not demonstrate that the compositional mechanism generalizes to higher-dimensional tasks. "

Reply: We have included a demonstration that our results hold in some higher dimensions, but that other aspects of task structure may be important to fully understand CL effectiveness. We tracked the relative difference in training time for factorized training and direct training in 3,5,10, and 20 dimensions. We also show that the trajectories encounter more saddles throughout an optimization in 3 dimensions. While we found that 3D and 5D systems still show improvement with factorized CL, 10D and 20D variants did not. We performed some preliminary investigation to see if the scaling of weight norms with dimensionality was an issue, but when accounting for this it did not show CL improvement. We are still investigating why results might be different for high-dimensional systems with the task set we have chosen. We reported all of these results in a main text figure 5.

2b. "In addition, the joint-training experiment initializes from parameters already constructed near an optimum, which limits its ability to probe learning dynamics from general starting conditions. Consequently, the current evidence does fully support authors assumptions."

Reply: We only initialized the recurrence near the ground truth value, but we placed the input and output weights far away from the optimum, at 50% of their optimal values. We chose this regime because jointly training with recurrence can be unstable, requiring parameter clamping and gradient clipping. We chose to stay in a regime that did not require this as managing instabilities of optimization was not our aim. Still, to address this concern, we have re-run the joint training simulations where initial recurrence is away from the optima and we used gradient clipping (norm < 1e4) and parameter clamping (recurrence between 0 and 1). We added these results to Appendix A6.5, showing that the key qualitative result still holds when initialized away from the optimum.

AuthorsOpenReview ↗

Reply to reviewer comments 2/2

  1. "3, The paper does not discuss [1]'s analysis of how task alignment and initialization affect transfer in deep linear networks. Could author compare the predicted transfer conditions in the identity recurrence limit and clarify which effects specifically require temporal recurrence? [1]. Andrew K. Lampinen and Surya Ganguli. “An Analytic Theory of Generalization Dynamics and Transfer Learning in Deep Linear Networks.” ICLR, 2019."

Reply: We have added Lampinen & Ganguli into the “related work” section and have compared our approach to it. In that work they studied linear feed-forward networks, and provided an expression for “time-to-criteria” for MSE losses in a student-teacher framework. Similar to us, they find an expression for single task training time under the core assumptions from Saxe et. al., 2014, and they also considered a multi-task setup. This work differs from ours in two important ways. First, the primary metric for success in the multi-task setting was generalization performance (i.e., test error) when noisy teachers generated test data, while we studied training time improvement . Second they studied multi-task training of two tasks simultaneously, while we studied task training in sequence. By training tasks together, the authors were able to consider how manipulations to rotational differences in the input SVD modes mattered, which was not accessible to us by training in sequence since it would break the factorized assumptions of Saxe et. al. 2014. This is why we focused on strength and temporal structure of task covariances. The authors investigated task similarity and training benefit in complementary, but different ways to our work.

  1. "The paper states that the rescaling symmetry guarantees (Appendix A.1). Gradient flow generally conserves a^2-c^2, while equality(a^2=c^2) requires balanced initialization. Can author state this assumption explicitly and explain how the training-time criterion changes for unbalanced initialization?"

Reply: We have clarified this symmetry condition and equality assumption in the appendix. Similar to Saxe et al (2014), failing to initialize with equal weights introduces a transient coupling term that dissipates throughout training. We had already mentioned that this initial condition was important in the main text formulation (Section 3), but, we have added a note of this distinction in the appendix.

Thank you for the clarifications and additional experiments. The correction of the zero-weight solution and the new initialization experiments address parts of my concerns.

However, the explanation based on avoiding saddles or slow points remains insufficiently supported in the full parameter space. I will keep my original score.

AuthorsOpenReview ↗

Characterization of saddle-like structure in full parameter space

We thank the reviews for the helpful comment about rigorously characterizing the slow points in the full parameter space. We have included further evidence that these points are indeed saddles in the factorized case, and that they become slow points with saddle-like curvature once compositional structure is added. We calculated the hessian spectrum and gradient norm while parametrically varying the strength of the compositional element of the task (i.e., the off-diagonal element of the output weight matrix). This captures the structure of the loss landscape in the full parameter space. These results have are in supplemental Section 12 (Fig. S12).

We first looked at how our theoretical saddles change as a function of this compositional strength. We see that when coupling strength is 0 (factorized case), the gradient norm is 0, and the hessian spectrum contains positive and negative eigenvalues, a true saddle in full parameter space. As compositional strength increases, this curvature persists, but the norm of the gradient slightly increases, suggesting a slow point. We see this for the saddles that are not the null solution, but the null solution maintains its strict saddle behavior. We want to emphasize that the gradient norm increase is still quite a small gradient; it is orders of magnitude larger than the initial gradient amplitudes seen in CL training (Fig. S11), and its magnitude comports the transient dips in gradient magnitude that we originally used to characterize slow points in the main text (Fig 4A, right).

However, we also sought to check if this slow point was obscuring a shift in saddle positioning once compositional structure is added. We performed a fixed-point finding search that minimized the kinetic energy on the loss landscape, beginning from the theoretical saddle location, but for the compositional task. We found that fixed points did indeed shift in the compositional case, with distinctly different losses than saddle locations from factorized theory. When we calculated their curvature, we found that they reflect saddle-like structure. An important note is that we could not optimize the kinetic energy to extremely low values (KE < 10^-14), which is possible with true saddles. We found that the KE was 10^-11 for one saddle, and 10^-9 for another. The judgement call about how weak of a gradient should be considered a minima is subjective, but previous work from Sussillo and Barack (2013) considered KE losses above 10^-4 as slow points. This places our new fixed points found through this minimization somewhere in the middle, thus we refer to them as slow points. Moreover, there is precedent for this type of effect of seeing slow points with saddle-like structure, which is akin to functional dynamical systems equivalences reported for computational dynamics elsewhere (e.g. Ságodi et al, Neurips 2024)

Finally, we show that a sample optimization trajectory for direct training has loss plateaus that correspond with skirting near these numerically located fixed points. This provides additional evidence that it is indeed the presence of these slow points, with saddle-like curvature in full parameter space, that are behind our CL results. Additionally, this is also correlated with traversing nearby the position of factorized saddles, which motivates us using them to assess performance in our 2D phase space plots in Figure 4.

We would like to end with a high level remark that an optimization’s progress can really only be slowed by two things: either it takes a longer trajectory, or it is hindered by a feature of the loss landscape such as fixed points and slow points. The first explanation is not true in our data: our CL optimizations overshoot target solutions, and traverse longer distances in parameter space. We have now given extensive evidence showing that it is the geometric properties of saddle-like points that are driving our results. If the reviewer has other possible explanations for us to account for, we will do our best to examine them in the time remaining.

Again, we thank the reviewer for the insightful comments, as it has substantially improved the rigor of our manuscript. If there are any other possible analyses or explanations that we can provide to improve our score, please let us know.

Reviewer_rCfVOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The article is difficult to read, so it is entirely possible that I didn't understand the main results. The authors extended the recent theory for linear RNNs (Proca et al. 2025) to determine which kinds of CL are good. In particular they study the speedup effects of CL in terms of equivalent initial conditions. They investigated factorized tasks in a more general case than in Proca et al, and what they called compositional tasks (although these do not really fall in the scope of linear RNN theory). They identified some general principles for which CL leads to speed improvements

Critical strengths

The Proce el al theory makes an unreasonable number of assumptions. The authors relaxed some of them and considered a broader class of tasks. They also identified a general principle that seems to be relatively simple. Finally, I appreciate the effort of linking CL speedups to the geometry of the loss.

Critical weaknesses

The analytical calculations are derived under a very large number of assumptions. In particular, one is particularly limiting: they are considering only the case in which the input and output matrices (called W_x and W_y) are trained, and not the recurrent weights. The theory is probably very challenging, but this is a major limitation. Finally, they studied what they called compositional tasks (tasks that simply combine different factorized modes). I'm not sure I would call them compositional, but anyway, it is not clear how much they add to the factorized ones, and also, they do not really have a theory for this case; the study is mostly empirical. The authors claim to have gained some insight into these cases from the theory, but it's not clear what that insight is.

Questions for authors

Make examples of tasks that could be described as factorized and compostional would be useful

3 comments
AuthorsOpenReview ↗

Reply to reviewer comments

We have addressed the reviewer concerns below. To ease discussion, we have reiterated and numbered each concern.

  1. "The analytical calculations are derived under a very large number of assumptions. In particular, one is particularly limiting: they are considering only the case in which the input and output matrices (called W_x and W_y) are trained, and not the recurrent weights. The theory is probably very challenging, but this is a major limitation."

Reply: These assumptions do limit the scope under which an analytical solution to learning dynamics and training time is tractable. This is precisely why we took steps to demonstrate that the trends in speed of training in that limited regime still held outside of the theory’s scope. It is true that we don’t have theoretical work for this case; instead we use an equally important approach of extensive numerical validation. We directly addressed the concern with new numerical results in Appendix A.6.5. These confirm that the regime where recurrence is fixed is not a strict limitation; rather, it allows us to connect our results to a rich body of work in feed-forward networks.

  1. "Finally, they studied what they called compositional tasks (tasks that simply combine different factorized modes). I'm not sure I would call them compositional,..."

Reply: The term compositionality is used broadly and there isn’t yet an agreed upon formal definition for a compositional task. Our composed tasks fall under the umbrella of “functional composition,” a definition used by Zhou et al (2024), in which a composed task c(x) can be expressed as a function of simpled tasks: c(x) = c(s_1(x), s_2(x), ..s_n(x), x). We have added this definition into the main text.

  1. " but anyway, it is not clear how much they add to the factorized ones, "

Reply: The compositional tasks add evidence that any intuitions or guiding principles found in factorized tasks can hold outside the formal scope of the theory. We chose compositional tasks as our primary use case because they are a common scenario in practice (see also text edits in sections 2 and 3).

  1. "and also, they do not really have a theory for this case; the study is mostly empirical. The authors claim to have gained some insight into these cases from the theory, but it's not clear what that insight is."

Reply: Unfortunately the formal theoretical scope of compositional tasks is limited. Despite this imperfect mathematical handle by using factorized tasks, though, we found through extensive empirical observations that the same trends broadly hold in the compositional setting. Since our original submission, our empirical work has enriched the intuitions driving this. By examining the individual contributions to losses (Supp. Fig 11) , we have learned that the non-composed channels of the task are main drivers of the loss. And through examining a 3D example of the task (Fig. 5), we found that more slow points were encountered, a geometric prediction made from factorized theory that appears important for compositional tasks.

  1. "Make examples of tasks that could be described as factorized and compositional would be useful"

Reply: We previously provided schematic examples of factorized and composed tasks in Figure 1c, and we also provided explicit, canonical examples of task parameters for each case in eqs. 4-5. To address the reviewers concern here, we have included a real-word example, and also included a primary reference for what constitutes a composed task (Zhou et al 2024).

I appreciate the authors' efforts to address my concerns. I agree with the authors that there isn’t yet an agreed upon formal definition for a compositional task. It is definitely helpful to have a clear definition of compositionality in the main text. The authors are obviously aware of the limitations of their analytical approach and performed extensive numerical validations (which are a fundamental component of the article). The new empirical work (Supp. Fig 11) is a valuable addition to the paper, especially for the new intuitions. The new version of the article is definitely stronger.

We thank the reviewer for the comments that increased the rigor of our work. We have now substantially improved our analyses in this work since the original submission, which we feel reflects a stronger claim that effective CL strategies have intuitive guiding principles from factorized learning theory, and that they function by avoiding saddle-like slow points on the loss landscape. We performed detailed characterization of loss contributions, performed additional analysis that include recurrence, performed a more careful theoretical and empirical characterization of fixed point structure, and showed how these results scale at higher dimensions. If there are any additional comments that we can address that will positively impact our score, please let us know.

Reviewer_8x9LOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work studied when pre-training on a task T1T_1 can reduce the total time required to learn a related second target task T2T_2 using in a linear RNN with fixed recurrent connectivity and trainable input and output weights. They start with a two-input example in which the task depends on a discounted running sum of each input stream, zi(t)=xi(t)+λizi(t−1)z_i(t)=x_i(t)+\lambda_i z_i(t-1) such that the network has two different discount factors, λ1\lambda_1 and λ2\lambda_2, and needs to learn how much of each input it should "route" through the two temporal filter and how it should combine the results at the resdout.

The authors first analyze "factorized" tasks in which each output depends simply on a separate filtered input, y^i=uizi\hat y_i=u_i z_i. In this setting, they calculate how each uiu_i changes during learning allowing for temporal correlations within each stream. This leads to a clear reason to switch tasks early, i.e, if the target asks for gzigz_i then training toward GziGz_i with G>g>0G>g>0 can bring uiu_i close to gg sooner than training toward gg directly. They then test a "compositional" task whose outputs are gz1gz_1 and g(z1+z2)g(z_1+z_2) and found that learning Gz1Gz_1 and Gz2Gz_2 separately first ends up requiring the fewest total training steps.

Critical strengths

I lean positive overall. I quite liked the intuition and reason it provides to stop pretraining early. For pretraining to be worthwhile, the time saved on T2T_2 has to exceed the time spent on T1T_1 and indeed their calculation give us the time spent on both tasks and tells us when that strategy actually pays for itself. The theory also leaves us with insightful explanation: for an input zz and desired output y∗y^*, the relation the network eventually at convergence is u∗=E[zy∗]/E[z2]u^*=\mathbb E[zy^*]/\mathbb E[z^2] but meanwhile it's growth at initialization depends on E[zy∗]\mathbb E[zy^*], implying that early learning and convergence is are governed by two different properties. This leads to a fairly specific prediction from the theory that i found interesting, which is the integration example in Fig. 2, where giving more weight to older inputs during pretraining can speed up learning a target that uses a shorter history.

The empirical comparison for me is between pretraining on G(z1,z2)G(z_1,z_2) and on G(z1,z1+z2)G(z_1,z_1+z_2) in the paper is quite good too, where teaching the network to report the two sums separately first leads to faster total training, which is notrivial and somewhat surprising and gives the compositional experiment much of its value.

Critical weaknesses

I didn't fully understand why teaching the two sums separately wins over pretraining on the scaled target in the compositional example, which for me is one of the more interesting results in the work. At large GG, Fig. 4C shows that detected saddle encounters during target training are rare for both the factorized curriculum G(z1,z2)G(z_1,z_2) and the scaled-target curriculum G(z1,z1+z2)G(z_1,z_1+z_2), yet their total training times in Fig. 3A are quite different. It makes sense that avoiding these slow regions can help explain why direct training takes longer. The authors also suggest that factorized pretraining leaves the weights closer to a target solution, although the explicit comparison there is with a permuted task. I was less clear on what explains the difference between the two curricula here. One possibility is that it is simply quicker to learn the two sums separately during T1T_1. It could also be that learning them separately leaves the network better prepared for T2T_2. Splitting the total time into pretraining and target training would show how much of the difference comes from each stage. If some of the advantage comes during T2T_2, I would also find it informative to compare how many steps each network still needs once both reach the same target error. That would show whether the factorized network continues to learn faster from that point onward. More generally, i was a bit unsatisfied with the theory and explanation behind the compositional case.

Questions for authors

Going back to my question on the critical weakness. Fig. 3A, where does teaching the two sums separately save time compared with pretraining on the scaled target? Are the sums quicker to learn on their own, or does the network also pick up their combination faster after the switch? Are there ways we can get more theoretical clarity on the compositional case? More broadly, if there are several ways to break a task into parts (apart from the one chosen the work), what should one look for in the data or early tasks to generate a curicullum?

3 comments
AuthorsOpenReview ↗

Reply to reviewer comments

We have addressed the reviewer concerns below. To ease discussion, we have reiterated and numbered each concern.

1a. "I didn't fully understand why teaching the two sums separately wins over pretraining on the scaled target in the compositional example, which for me is one of the more interesting results in the work. ... I was less clear on what explains the difference between the two curricula here. One possibility is that it is simply quicker to learn the two sums separately during (T_1). It could also be that learning them separately leaves the network better prepared for (T_2). Splitting the total time into pretraining and target training would show how much of the difference comes from each stage. If some of the advantage comes during (T_2), "

Reply: We have partitioned the training time between Task1 and Task2 for factorized and scaled training when the scaling factor is large (c=15.0). Additionally, we also show the contribution to the loss from each output channel over learning. In factorized CL, channels can optimize independently, and we find that each channel optimizes at a similar rate. When training on the target task, we see a quick convergence of the composed channel (channel 2, red), but interestingly it is the independent, factorized channel that dominates the remainder of the optimization. In direct training, the loss for the composed channel is initially larger, but again we see a quick convergence of this loss, and that the factorized channel dominates the optimization. We believe that this effect is mitigated in CL because the initial stages of training must not simultaneously maintain the contribution from the composed channel. The scaled target task at the strength C=15.0 did not have a training benefit, and indeed we see that it had the same issues that direct training had. We have added this result in Appendix A.6.11. Lastly, Figure 4E addressed the reviewer’s question about scaled-target vs. task-permuted training. It shows a 2D state-space version for strong weights (C=15.0), where factorized training was beneficial but scaled target and task-permuted CL did not help. In that plot the weights just before training on the target task are the same distance from the target, which are further away than factorized CL weights.

1b: "I would also find it informative to compare how many steps each network still needs once both reach the same target error. That would show whether the factorized network continues to learn faster from that point onward. "

Reply: The factorized network does learn much faster, which can be shown from the data mentioned above (Appendix A.6.11). The second task for factorized CL starts at several orders of magnitude larger than initial loss values for direct training, yet it optimizes in substantially fewer time steps (note: these results are on a log scale)

2: "More generally, i was a bit unsatisfied with the theory and explanation behind the compositional case."

Reply: Unfortunately the formal theoretical scope of compositional tasks is limited. Despite this imperfect mathematical handle by using factorized tasks, we found through extensive empirical observations that the same trends broadly hold in the compositional setting. Since our original submission, we have been able to enrich the intuitions behind this. By examining the individual contributions to losses (Supp. Fig 11) , we have learned that the non-composed channels of the task are main drivers of the loss. And through examining a 3D example of the task (Fig. 5), we found that more slow points were encountered (e.g., the influence of a nearby saddle), a geometric prediction made from factorized theory that appears important for compositional tasks.

3: "More broadly, if there are several ways to break a task into parts (apart from the one chosen the work), what should one look for in the data or early tasks to generate a curicullum?"

Reply: This is a great question, and we now have expanded upon it in the discussion. Since we found that factorized training was an effective curriculum task, we highlight that a primary issue is to identify those subtasks in a principled way. One approach is to use a normative account of the task to identify primitive tasks, an approach suggested by Hocker et. al., 2025. Alternatively, a data-driven approach using unsupervised learning could empirically estimate the input-target covariance, which could inform the primitive tasks.

Thank you for your comments, I am broadly satisfied with them, but while the answers clarify, they don't necessarily strnegthen the paper more. So, I am keeping my score.

AuthorsOpenReview ↗

Outstanding analyses to improve score?

We thank the reviewer for the comments that increased the rigor of our work, and are happy that they are broadly satisfied with our detailed answers to their comments. Since they felt that the updated results did not warrant changing our score, we ask that if there are any additional comments that we can address that will positively impact our score, please let us know.

We have now substantially improved our analyses in this work since the original submission, which we feel reflects a stronger claim that effective CL strategies have intuitive guiding principles from factorized learning theory, and that they function by avoiding saddle-like slow points on the loss landscape. We performed detailed characterization of loss contributions, performed additional analysis that include recurrence, performed a more careful theoretical and empirical characterization of fixed point structure, and showed how these results scale at higher dimensions.

Reviewer_UjguOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies a toy setting of linear RNNs. Dynamics of learning for certain structured targets (with factorized forms and compositional forms) are derived. These dynamics are used to study the effect of training first on a Task 1 and subsequently training on a Task 2.

The amount of training time with gradient flow is computed for the different tasks. In particular, the main object of study is whether pretraining on task 1 and then fine-tuning on task 2 takes less time than just training on task 2.

Critical strengths

The paper provides a rigorous, clean mathematical analysis of a toy setting.

Critical weaknesses

While I do not doubt that the mathematical derivations are correct, there is a fundamental issue with using training time with gradient flow as a way to measure complexity of learning. Gradient flow can be time-reparametrized without changing the trajectories, so it is not a true measure of the complexity of learning (rather just a proxy that depends on the particular parametrization). Thus, I am not convinced by these results as giving a rigorous view into how much pretraining on a given task helps. I would be more convinced by finite-sample analysis of SGD (or, better yet, Adam) with optimally-chosen learning rates. (Of course, such analyses are

It is unclear to me how generalizable the insights are. For instance, consider the insight that larger weights after pretraining help, which is the "be loud" part of the title. This seems (A) to rely on the fact that gradient flow is being analyzed, and thus a larger initial weight after Task 1 essentially entails a larger effective step size, which means faster convergence when fine-tuning to Task 2. Additionally, (B) it is unclear to me whether this insight is affected in practice by things such as layernorm, which will serve to standardize the relative magnitudes involved.

Overall, the other insight, "be relevant" (which is that task 1 should be relevant to task 2) just seems to be confirmation of an intuition that people already have.

Questions for authors

-- typos: cmore, asses, difficultly

Please address my comments on the gradient flow and generalizability of the insights.

1 comment
AuthorsOpenReview ↗

Reply to reviewer comments

We have addressed the reviewer concerns below. To ease discussion, we have reiterated and numbered each concern.

  1. "Complexity of learning. While I do not doubt that the mathematical derivations are correct, there is a fundamental issue with using training time with gradient flow as a way to measure complexity of learning. Gradient flow can be time-reparametrized without changing the trajectories, so it is not a true measure of the complexity of learning (rather just a proxy that depends on the particular parametrization). Thus, I am not convinced by these results as giving a rigorous view into how much pretraining on a given task helps. I would be more convinced by finite-sample analysis of SGD (or, better yet, Adam) with optimally-chosen learning rates. (Of course, such analyses are."

Reply: The reviewer is right that generally speed of learning as a comparison metric is usually confounded by choices of hyperparameters or optimization schemes. Sample complexity would admittedly be cleaner, but it is intractable for the setup considered here. We controlled for this concern explicitly in our study by using the same optimization strategy (gradient descent in a specific model) and hyperparameters (learning rate) across all comparisons. We included additional numerical confirmation that the conclusions are invariant to the choice of learning rate (for sensible choices, see new Supplemental Fig S9). Importantly, we would like to emphasize that it is the geometry of the loss landscape(s) that dictates the fundamental effectiveness of different curricula, not the tool used to traverse it.

2a. "It is unclear to me how generalizable the insights are. For instance, consider the insight that larger weights after pretraining help, which is the "be loud" part of the title. This seems (A) to rely on the fact that gradient flow is being analyzed, and thus a larger initial weight after Task 1 essentially entails a larger effective step size, which means faster convergence when fine-tuning to Task 2. "

Reply: Increased effective step size (i.e. larger gradients) occurs at the beginning of training, when errors are large, phenomenon exacerbated when starting with large weights. We see this in our curricula, which allows them to take initially larger steps on the loss landscape (new Suppl. Figure S10). However, if this were the primary reason behind the pretraining’s success, then any large weight initial conditions would impart the same benefit. We do not see this. We reported a useful control for this scenario in Figure 3: randomly chosen tasks with the same scale do not show a training speed; nor do tasks with the same norm, but with permuted entries that make them misaligned to the target task. This points to a second critical mechanism, linked geometrically to avoiding specific regions of the loss space that have small gradients.

2b. "Additionally, (B) it is unclear to me whether this insight is affected in practice by things such as layernorm, which will serve to standardize the relative magnitudes involved."

Reply: This system is a one-layer network, and layer normalization techniques are only relevant for deep networks. Our framework intentionally used a shallow network to isolate the core effects that drive CL success without needing to address the many techniques required to handle issues in deep networks

  1. "Overall, the other insight, "be relevant" (which is that task 1 should be relevant to task 2) just seems to be confirmation of an intuition that people already have."

Reply: While the general intuition that pretraining tasks need to be useful for the target is uncontroversial, that statement can quickly get tautological. What we do here is to provide a mathematically precise definition of across-task relevance, prove formally and numerically the consequences of it for learning speed, and extract a common principle of compositional pretraining that can inspire more complex use cases in practice. While our theory verifies several commonly held pretraining intuitions, it also provides examples where common intuition is wrong: the belief that pretraining tasks should be trained to convergence is misguided, and the relationship between pretraining and target tasks dictates stopping criteria. We also provide a heuristic for when to stop in our discussion section to make this insight actionable in practice. More generally, we believe that mathematical precision in simple models can reinforce and refine common sense intuitions ML practitioners may have about what is needed for good pretraining, making our results useful and relevant for the ICLR community.

Comments on the paper

1 comment
Area_Chair_c2vzOpenReview ↗

Discussion Period

Dear Reviewers,

Thank you for your reviews!

The authors have now submitted their responses. Please read through their rebuttal, acknowledge their reply, and follow up with them if any of your concerns haven't been fully addressed or if you have additional questions.

Please make sure to engage in the discussion before the deadline on November 18 (AoE).

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.