- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work studies when simpler-task pretraining accelerates target-task training in linear RNNs. Extending existing learning-dynamics analysis to temporally structured task covariances with fixed recurrent weights, the authors derive speedup conditions for factorized, geometrically aligned tasks under proper initialization, governed by covariance strength and pretraining accuracy.
Evaluating multiple curricula on a 2D compositional task, they find sufficiently scaled factorized pretraining reduces training time and solution variance. The results are explained via the saddle avoidance, with supplementary experiments covering nonlinear activations, mismatched task geometry, and trainable recurrence.
Critical strengths
- the paper analytical treatment of total training time.
Equations 9–10 include the costs of pretraining when comparing a curriculum with direct target training which makes the analysis relevant to actual optimization savings. Although its conclusions depend on restrictive assumptions about task geometry and initialization.
- Controlled comparisons of curriculum structure.
The compositional experiments evaluate factorized tasks, scaled targets, target permutations, low-rank tasks and random tasks at varying amplitudes. These controls suggest the observed speedup cannot be attributed to weight magnitude alone.
Critical weaknesses
- The central geometric explanation needs stronger verification
In the paper, section 4.2 describes the zero-weight solution as a “local maximum”. However, for a nontrivial factorized mode, however, the loss near the origin contains a term proportional to which aligned and oppositely signed perturbations respectively decrease and increase the loss and made the origin a saddle in the unrestricted parameter space.
Moreover, loss plateaus, gradient norms, and a two-dimensional PCA slice do not establish that compositional trajectories avoid saddles in the full parameter space. Since saddle avoidance is the primary mechanistic explanation for the reported behavior, these issues affect the interpretation itself, not merely the terminology.
- The compositional evidence remains narrow
The compositional study is restricted to two hidden units and two input/output channels (Appendix A.4). Although the supplementary ReLU and jointly trained recurrence experiments extend portions of the covariance analysis, they do not demonstrate that the compositional mechanism generalizes to higher-dimensional tasks. In addition, the joint-training experiment initializes from parameters already constructed near an optimum, which limits its ability to probe learning dynamics from general starting conditions. Consequently, the current evidence does fully support authors assumptions.
Questions for authors
1, See Critical Weakness 1
2, See Critical Weakness 2
3, The paper does not discuss [1]'s analysis of how task alignment and initialization affect transfer in deep linear networks. Could author compare the predicted transfer conditions in the identity recurrence limit and clarify which effects specifically require temporal recurrence?
4, The paper states that the rescaling symmetry guarantees (Appendix A.1). Gradient flow generally conserves , while equality requires balanced initialization. Can author state this assumption explicitly and explain how the training-time criterion changes for unbalanced initialization?
[1] Andrew K. Lampinen and Surya Ganguli. “An Analytic Theory of Generalization Dynamics and Transfer Learning in Deep Linear Networks.” ICLR, 2019.
4 comments
Reply to reviewer comments 2/2
- "3, The paper does not discuss [1]'s analysis of how task alignment and initialization affect transfer in deep linear networks. Could author compare the predicted transfer conditions in the identity recurrence limit and clarify which effects specifically require temporal recurrence? [1]. Andrew K. Lampinen and Surya Ganguli. “An Analytic Theory of Generalization Dynamics and Transfer Learning in Deep Linear Networks.” ICLR, 2019."
Reply: We have added Lampinen & Ganguli into the “related work” section and have compared our approach to it. In that work they studied linear feed-forward networks, and provided an expression for “time-to-criteria” for MSE losses in a student-teacher framework. Similar to us, they find an expression for single task training time under the core assumptions from Saxe et. al., 2014, and they also considered a multi-task setup. This work differs from ours in two important ways. First, the primary metric for success in the multi-task setting was generalization performance (i.e., test error) when noisy teachers generated test data, while we studied training time improvement . Second they studied multi-task training of two tasks simultaneously, while we studied task training in sequence. By training tasks together, the authors were able to consider how manipulations to rotational differences in the input SVD modes mattered, which was not accessible to us by training in sequence since it would break the factorized assumptions of Saxe et. al. 2014. This is why we focused on strength and temporal structure of task covariances. The authors investigated task similarity and training benefit in complementary, but different ways to our work.
- "The paper states that the rescaling symmetry guarantees (Appendix A.1). Gradient flow generally conserves a^2-c^2, while equality(a^2=c^2) requires balanced initialization. Can author state this assumption explicitly and explain how the training-time criterion changes for unbalanced initialization?"
Reply: We have clarified this symmetry condition and equality assumption in the appendix. Similar to Saxe et al (2014), failing to initialize with equal weights introduces a transient coupling term that dissipates throughout training. We had already mentioned that this initial condition was important in the main text formulation (Section 3), but, we have added a note of this distinction in the appendix.
Thank you for the clarifications and additional experiments. The correction of the zero-weight solution and the new initialization experiments address parts of my concerns.
However, the explanation based on avoiding saddles or slow points remains insufficiently supported in the full parameter space. I will keep my original score.
Characterization of saddle-like structure in full parameter space
We thank the reviews for the helpful comment about rigorously characterizing the slow points in the full parameter space. We have included further evidence that these points are indeed saddles in the factorized case, and that they become slow points with saddle-like curvature once compositional structure is added. We calculated the hessian spectrum and gradient norm while parametrically varying the strength of the compositional element of the task (i.e., the off-diagonal element of the output weight matrix). This captures the structure of the loss landscape in the full parameter space. These results have are in supplemental Section 12 (Fig. S12).
We first looked at how our theoretical saddles change as a function of this compositional strength. We see that when coupling strength is 0 (factorized case), the gradient norm is 0, and the hessian spectrum contains positive and negative eigenvalues, a true saddle in full parameter space. As compositional strength increases, this curvature persists, but the norm of the gradient slightly increases, suggesting a slow point. We see this for the saddles that are not the null solution, but the null solution maintains its strict saddle behavior. We want to emphasize that the gradient norm increase is still quite a small gradient; it is orders of magnitude larger than the initial gradient amplitudes seen in CL training (Fig. S11), and its magnitude comports the transient dips in gradient magnitude that we originally used to characterize slow points in the main text (Fig 4A, right).
However, we also sought to check if this slow point was obscuring a shift in saddle positioning once compositional structure is added. We performed a fixed-point finding search that minimized the kinetic energy on the loss landscape, beginning from the theoretical saddle location, but for the compositional task. We found that fixed points did indeed shift in the compositional case, with distinctly different losses than saddle locations from factorized theory. When we calculated their curvature, we found that they reflect saddle-like structure. An important note is that we could not optimize the kinetic energy to extremely low values (KE < 10^-14), which is possible with true saddles. We found that the KE was 10^-11 for one saddle, and 10^-9 for another. The judgement call about how weak of a gradient should be considered a minima is subjective, but previous work from Sussillo and Barack (2013) considered KE losses above 10^-4 as slow points. This places our new fixed points found through this minimization somewhere in the middle, thus we refer to them as slow points. Moreover, there is precedent for this type of effect of seeing slow points with saddle-like structure, which is akin to functional dynamical systems equivalences reported for computational dynamics elsewhere (e.g. Ságodi et al, Neurips 2024)
Finally, we show that a sample optimization trajectory for direct training has loss plateaus that correspond with skirting near these numerically located fixed points. This provides additional evidence that it is indeed the presence of these slow points, with saddle-like curvature in full parameter space, that are behind our CL results. Additionally, this is also correlated with traversing nearby the position of factorized saddles, which motivates us using them to assess performance in our 2D phase space plots in Figure 4.
We would like to end with a high level remark that an optimization’s progress can really only be slowed by two things: either it takes a longer trajectory, or it is hindered by a feature of the loss landscape such as fixed points and slow points. The first explanation is not true in our data: our CL optimizations overshoot target solutions, and traverse longer distances in parameter space. We have now given extensive evidence showing that it is the geometric properties of saddle-like points that are driving our results. If the reviewer has other possible explanations for us to account for, we will do our best to examine them in the time remaining.
Again, we thank the reviewer for the insightful comments, as it has substantially improved the rigor of our manuscript. If there are any other possible analyses or explanations that we can provide to improve our score, please let us know.
Reply to reviewer comments 1/2
We have addressed the reviewer concerns below. To ease discussion, we have reiterated and numbered each concern.
1a. "The central geometric explanation needs stronger verification. In the paper, section 4.2 describes the zero-weight solution as a “local maximum”. However, for a nontrivial factorized mode, however, the loss near the origin contains a term proportional to which aligned and oppositely signed perturbations respectively decrease and increase the loss and made the origin a saddle in the unrestricted parameter space. "
Reply: The reviewer is correct that the zero-weight solution is not a local optimum. We numerically calculated the hessian of this solution and it is indeed a saddle. We had not fully appreciated the structure of the null solution, and thank the reviewer for the careful reading of our derivation. We have updated it in the main text.
1b. "Moreover, loss plateaus, gradient norms, and a two-dimensional PCA slice do not establish that compositional trajectories avoid saddles in the full parameter space. Since saddle avoidance is the primary mechanistic explanation for the reported behavior, these issues affect the interpretation itself, not merely the terminology."
Reply: The optimizations do not perfectly traverse through saddles, this is true. The norm of the gradient never fully diminishes. We believe this is because introducing compositional structure altered the nature of the fixed point. We had previously reported the norm of the gradients in the 2D PCA space in Supp. Fig. 8, where you can see clear saddle structure in the factorized setting that is smoothed out in the compositional case, suggesting a change in the fixed points’ characterization to a slow point. We have changed the language in the main text to reflect this point.
2a. "The compositional evidence remains narrow. The compositional study is restricted to two hidden units and two input/output channels (Appendix A.4). Although the supplementary ReLU and jointly trained recurrence experiments extend portions of the covariance analysis, they do not demonstrate that the compositional mechanism generalizes to higher-dimensional tasks. "
Reply: We have included a demonstration that our results hold in some higher dimensions, but that other aspects of task structure may be important to fully understand CL effectiveness. We tracked the relative difference in training time for factorized training and direct training in 3,5,10, and 20 dimensions. We also show that the trajectories encounter more saddles throughout an optimization in 3 dimensions. While we found that 3D and 5D systems still show improvement with factorized CL, 10D and 20D variants did not. We performed some preliminary investigation to see if the scaling of weight norms with dimensionality was an issue, but when accounting for this it did not show CL improvement. We are still investigating why results might be different for high-dimensional systems with the task set we have chosen. We reported all of these results in a main text figure 5.
2b. "In addition, the joint-training experiment initializes from parameters already constructed near an optimum, which limits its ability to probe learning dynamics from general starting conditions. Consequently, the current evidence does fully support authors assumptions."
Reply: We only initialized the recurrence near the ground truth value, but we placed the input and output weights far away from the optimum, at 50% of their optimal values. We chose this regime because jointly training with recurrence can be unstable, requiring parameter clamping and gradient clipping. We chose to stay in a regime that did not require this as managing instabilities of optimization was not our aim. Still, to address this concern, we have re-run the joint training simulations where initial recurrence is away from the optima and we used gradient clipping (norm < 1e4) and parameter clamping (recurrence between 0 and 1). We added these results to Appendix A6.5, showing that the key qualitative result still holds when initialized away from the optimum.