- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies auxiliary learning in a teacher–student framework, deriving online SGD dynamics and a leading-order expression for the steady-state generalization error of linear networks. It extends the analysis to nonlinear activations through a reduced Lyapunov equation, with numerical simulations supporting its predictions of how loss weights, readout correlations, and label noise affect generalization error under the fixed-readout setting.
Critical strengths
- The paper develops a clean, tractable teacher–student framework for auxiliary learning, with an explicit ODE description of online SGD and steady-state analyses covering both linear and nonlinear activations. The theoretical analysis yields concrete results, including a closed-form leading-order error for linear networks and, for nonlinear activations, a reduction of the stationary fluctuation analysis to a Lyapunov equation.
- The presentation is clear, and the analytical predictions are supported by comparisons with ODE integration and SGD simulations in the regime studied.
Critical weaknesses
-
The connection between the theoretical findings and the auxiliary-learning phenomena motivating the paper remains insufficiently established. The framework provides a tractable account of stationary prediction error, but it is less clear what these findings tell us about when auxiliary supervision helps or harms the main task through learning a shared representation. Both targets are generated from the same teacher features and admit a common realizable optimum, while the main analysis fixes the readouts to their teacher values. The experiments largely validate the theory within these assumptions, leaving unclear which weighting laws and error relations survive when the readouts are learned jointly, task demands compete for shared capacity, or training repeatedly uses finite data. A modest controlled check relaxing one of these assumptions, or a substantive analysis explaining which findings should persist and why, would help bridge the model to the broader auxiliary-learning questions raised in the introduction. Successful extrapolation or a complete theory is not required; identifying where the regularities change would itself clarify the contribution’s scope and significance.
-
The introduction could state the main findings and their significance more concretely. The abstract and introduction explain what is analysed and say that task correlations and label noise determine the auxiliary benefit, but do not make the resulting regularities explicit. For example, highlighting the correlation-independent optimal auxiliary weight in the linear setting or the weighted-error identity, with their conditions, would give readers a clearer understanding of what has been discovered. The revision should foreground a few principal findings and explain what they add to the understanding of auxiliary learning, rather than primarily describing the analytical framework and technical outputs. This is primarily a presentation concern.
Questions for authors
The main requests are in the Weaknesses section. The following clarifications would also be helpful; a discussion would suffice, with additional experiments optional.
-
Role of the learning rate. Appendix C identifies an effective learning rate and label noise for , while Appendix E gives at , where the improvement comes from reducing the main-task learning rate. How should the auxiliary benefit be interpreted relative to single-task training with a reasonably tuned learning rate, particularly in terms of the error–convergence-speed trade-off or performance under a fixed training budget? This does not question the validity of the fixed-learning-rate comparisons, but would help clarify the significance of the reported gains.
-
Positioning relative to closely related work. For a single output, at a realizable optimum with independent homoscedastic label noise, the gradient-noise covariance is proportional to the Hessian, which gives a simple solution to the leading-order stationary covariance equation. Could the authors clarify the specific advance of Eq. (22), , relative to classical LMS misadjustment results and the stationary analysis of Mandt et al. (2017)? In particular, is the advance its explicit evaluation for general activations in the teacher–student setting? It would also help to discuss Mori et al. (ICLR 2025), which studies task-selection protocols in closely related teacher–student models for multi-task and continual learning.
References:
- B. Widrow, J. M. McCool, M. G. Larimore, and C. R. Johnson, “Stationary and nonstationary learning characteristics of the LMS adaptive filter,” Proceedings of the IEEE, 64(8):1151–1162, 1976.
- F. Mori, S. Sarao Mannelli, and F. Mignacco, “Optimal Protocols for Continual Learning via Statistical Physics and Control Theory,” ICLR 2025 (arXiv:2409.18061).
3 comments
Reviewer Fy32: response to rebuttal — score raised to weak accept
Thank you for the response and the revision.
W1. Section 3.4 and Figure 5 are the controlled check I had in mind. For , the fixed-readout prediction still agrees with the experiment when the second layer is trained. For 1-homogeneous activations, the error continues to fall as norm transfers between layers, and the results agree with the theory only after rescaling by . The larger- results in Figure 4 are also useful.
Two details from the discussion should be included in the paper:
- The response identifies finite training data and competing demands for shared capacity as limitations. I could not find these in the revised PDF. Please state them explicitly, along with the explanation given in the reply to Reviewer Grnh: in the realizable online setting studied here, the auxiliary task improves generalization through noise averaging and an effective change in learning rate.
- Figure 5 shows a single run at one hyperparameter setting. Please make that clear in the caption, or add a few seeds or a second value of .
W2. The revised introduction now motivates the work in terms that match the setting analyzed. This resolves my concern. One typo remains on p. 1: “the optimal weigh and that the optimal auxiliary weight.”
Q1. Appendix E answers my question. The auxiliary head converges faster and reaches a lower steady-state error. These results are sufficient for my assessment, and I do not need an additional single-task curve with a tuned learning rate.
Q2. The positioning relative to Mandt et al. (2017) and Mori et al. (2025) is now clear. Please add one sentence relating Eq. (22) to the classical LMS misadjustment result, citing Widrow et al. (1976).
The revision addresses my main concerns, and I am raising my score to weak accept.
We thank the reviewer for the follow-up and for supporting acceptance. We updated the PDF with the requested changes and renamed Section 4 to "Future Work and Limitations" to better reflect its content.
We thank the Reviewer for their constructive comments on our manuscript. We addressed the two critical weaknesses by adding simulations with a learnable second layer (in the manuscript, Sec. 3.4 and Fig. 5) and by expanding the introduction for clarity. We also added App. E and Fig. S4 showing how the additional head speeds up convergence across βs, as well as performance. We also added Mori et al. (2025) to the related work.
To address these concerns and expand the scope of our results, we relaxed the assumptions of frozen readouts, and we added Sec. 3.4 and Fig. 5 to show what happens when training the second layer. Networks with non-homogeneous activations such as erf converge to the frozen 2nd-layer Lyapunov theory, while networks with ReLU and linear activations keep improving without plateauing. This is because if an activation function is 1-homogeneous (f(cx)=cf(x)), there is a degeneracy of solutions. SGD then moves the weight norm from the second to the first layer, improving generalization by effectively decaying its learning rate. But rescaling this error by the second layer weights once again recovers our theoretical prediction.
As for the weighting laws, we updated Fig. 4 to show results with different K. With larger Ks, optimal β∗s are closer to the optimal 2−1 of the linear model. We were not able to derive the analytical limit for K→∞, if such a limit exists, but it could be a future direction to explore. Training with a finite dataset is another interesting direction, which we believe to be outside the scope of this analysis. We note DMFT as a promising route (for example, Bordelon&Pehlevan, 2023). In general, “competing demands for shared capacity” is not covered and is a stated limitation of our work.
We agree with the reviewer about the introduction, and we have expanded it, hopefully making our paper's findings clearer.
Q1. To answer this question, we added App. E and Fig. S4. They show how convergence time changes with β (with other hyperparameters fixed, and α=1). We consistently observe that having an additional head speeds up convergence, and the gain is larger with smaller ρs and larger βs. With smaller learning rates, models require more steps to converge, and thus training a single-head model to match the generalization error of a multi-head model with a well-tuned β also requires significantly more steps. This is important because in online learning, convergence time also indicates how much data is needed to converge.
Adding to this, the results on the trainable second layer we discussed above show that the effective learning rate of the network can change dynamically directly from SGD, without any scheduling. In a way, SGD keeps the same learning rate until convergence, and then starts diminishing it by moving the weight norm from the second to the first layer. Possibly, a learning rate schedule might perform even better, but deriving it analytically is left to future research.
Q2. We agree with the observation on the Hessian. While there are similarities between our results and Mandt et al. (2017), which partially inspired our approach, there are important differences. From a technical angle, their results rely on using mini-batches and invoking the central limit theorem, and the noise comes from mini-batches resampled from a fixed, finite dataset. In our setting, we specialize in a solvable model trained with online SGD and noise coming directly from the teacher. Thus, the Lyapunov equation is solvable a priori because the thermodynamic limit reduces our problem to O(K2) variables. But our goals are also different: they focus on tuning SGD parameters from a Bayesian point of view and using SGD as a variational EM algorithm, while we want to solve our models analytically to better understand their behaviours.
We also thank the reviewer for pointing out that we missed Mori et al. (2025) among the previous literature. We have corrected this. The main difference in our work is that we train tasks in parallel, while they use continual learning, where tasks alternate. They choose a teacher with only one neuron for each task. They then introduce a task-similarity parameter γ between the 1st-layer weights, while our ρ is between the 2nd-layer weights. But the biggest difference is that they focus on convergence and forgetting, two dynamical aspects of training, while we study the behaviour of the network already at the steady-state solution.