- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The authors seek to better understand multitask learning in recurrent neural networks. They employ an impressively large task array (100 cognitive tasks), and use variations of an almost-linear RNN architecture (AL-RNN) to investigate the effects of hierarchical structure, nonlinearity, and task feature embedding on multitask learning. The paper makes several interesting contributions, including demonstrating that a hierarchical architecture can successfully solve a large battery of tasks despite generating task-specific RNN parameters from a low-dimensional embedding through a shared linear hyper network. This architecture maintains task-specific flexibility while constraining recurrent parameters to a common subspace, and outperforms flat jointly trained and individually trained networks, as well as LSTM, GRU, and vanilla RNN controls. The authors demonstrate that jointly solving tasks, even those which are individually linearly solvable, requires additional nonlinear capacity, highlighting potential mechanisms by which recurrent networks are able to do multitask learning. Furthermore, the paper employs three complementary methods of task similarity (learned task embeddings, piecewise linear activation region usage, and recurrent state geometry) to look at how task representations and dynamics organise. They relate these metrics to learning transferability, demonstrating that tasks which are more isolated within the similarity spaces generally provide less effective initialisations for learning other tasks. This paper provides evidence that structured low-dimensional parameter sharing can improve recurrent multitask learning, and offers interesting insights into task embeddings and transferability.
Critical strengths
S.i) Scale and scope is impressive - this paper studies multitask recurrent learning using an unusually large battery of 100 cognitive tasks, extending previous work that typically uses smaller and more structured task sets. This is a really cool addition to the literature base, and the authors conduct very nice examinations of how representational similarity influences task learning transfer.
S.ii) Strong architectural controls. This paper employs the AL-RNN in a hierarchical, flat, and individual manner, but also makes control comparisons with other recurrent architectures such as LSTM, GRU, and vanilla RNNs. The findings are made stronger by this.
S.iii) Interesting findings. I found the results particularly impressive and well structured. In particular, the relationship between individual task complexity and joint solvability, the range of similarity measures and investigation of these both across seeds and architectures, and the authors approach to ensuring robustness in results were appreciated. Furthermore, the hierarchical parameter sharing results are very interesting - that very few task-feature dimensions are sufficient to approach ceiling-level performance is suggestive that the hierarchical AL-RNN architecture can support diverse tasks through a very low-dimensional subspace is very impressive.
Critical weaknesses
W.i) The interpretation of sharing in the hierarchical architecture needs greater clarification and precision. Given that each task receives a distinct set of recurrent parameters, although these are generated from the same low- dimensional subspace, some claims concerning shared dynamics and shared recurrent backbone appear stronger than what is supported by the architecture.
Questions for authors
I found the paper really interesting and am broadly positive about its contributions. My main concerns relate to the interpretation of shared recurrent dynamics and transfer/generalisability results. I believe that the paper is largely very sound, so these concerns are suggestions to strengthen it.
Major concerns:
M.i) My primary conceptual concern relates to the interpretation of what is shared within the hierarchical architecture.
a) Lines 125-126 state that the framework should allow new tasks to be learned “without fitting an independent network for each task or modifying the shared dynamics.” This distinction should be clarified further. As I understand, the hierarchical AL-RNN architecture generates a separate set of recurrent parameters for each tsk, with these parameters generated from the shared low- dimensional subspace from G. There is clearly meaningful parameter sharing and reduced task-specific degrees of freedom, but each task is solved by a differently parameterised recurrent network. It would be good for the authors to clarify what they mean by avoiding independent networks or relation to shared dynamics.
b) More generally, terms such as “shared recurrent backbone”, “shared recurrent dynamics”, and “shared recurrent model” appear throughout the paper, and are potentially misleading when referring to the hierarchical AL- RNNs. This model produces task-specific networks, which do not share recurrent weights or dynamics, what is shared is the parameter generation subspace. While this does provide an important form of sharing, the authors must make clear the distinction between their model sharing recurrent parameterisation subspace and sharing recurrent dynamics, particularly when interpreting the mechanisms of task separation and reuse.
c) The clarity should be improved when discussing comparisons across tasks for the hierarchical AL-RNNs. Since the hierarchical architecture generates different recurrent parameters for each task, it is not immediately clear how geometric similarity between task-specific state spaces should be interpreted. Although the recurrent units have corresponding indices across tasks, their functional roles may differ. Could the authors clarify how this affects the interpretation, and whether shared parameterisation provides sufficient alignment for comparisons? Or alter these analyses to place task-specific networks into a fully shared subspace for comparisons? In particular, it would be helpful to distinguish similarity in state-space geometry from evidence for shared computational mechanisms. Overall, the paper would benefit a lot from distinguishing better between sharing a recurrent parameterisation subspace and sharing a recurrent substrate, particularly when interpreting the results.
M.ii) What is the relevance of varying the level of nonlinearity? I saw more of the motivation with regards to independently varying nonlinear capacity and latent dimensionality, but I think the primary motivation for using the AL-RNN is unclear, and the results concerning the effects of nonlinearity across architectures would benefit from further interpretation. The observation that performance in the flat model peaks with sparse nonlinearity vs hierarchical model benefits from full nonlinearity is interesting, but not discussed in much detail or interpreted. Could the authors elaborate on why additional nonlinearity appears detrimental to the flat architecture but beneficial for the hierarchical architecture? It’s especially interesting considering the idea that the flat model must encode all of the tasks in different regions of the recurrent space and switch between them using nonlinearity, but then seems to impair performance when there is more nonlinearity. It would be particularly interesting to understand whether this reflects differences in optimisation difficulty, task interference, or the dynamical mechanisms used to separate tasks.
M.iii) I found the training protocols difficult to follow from the main text and appendix A.2, I think further clarification is necessary to improve reproducibility. In particular:
a) Trials are sampled in batches of 64 with balanced task representation across the training pool, but it’s unclear whether individual minibatches are task balanced or whether tasks are sampled randomly with balancing occurring across the full training procedure.
b) Please clarify the number of random seeds used across architectures and analyses, and whether this varies between experiments.
c) Please provide detail on how parameter budgets are calculated and matched across architectures, particularly for the hierarchical AL-RNNs (i.e. it’s unclear whether the counted parameters are from G and p only - and how they are calculated).
M.iv) These points largely relate to the transfer and leave-one-out experiments, which I found to be very interesting, but believe that they lack some critical further investigation and controls which would strengthen the results.
a) The results relating task isolation to transferability are interesting, but I think they would benefit from further behavioural or computational interpretation. Approximately 30% of source–target pairs exhibit negative transfer relative to random initialisation. Are there identifiable task characteristics associated with these pairs, or with tasks that consistently act as poor transfer sources? Similarly, it would be useful to identify which tasks are most isolated in the similarity spaces shown in Figure 11, and whether their isolation corresponds to particular computational requirements. Finally, are tasks that perform poorly as initialisations in the pairwise transfer experiments also poor sources for leave- one-out adaptation? Exploring these relationships could help connect the learned similarity measures to more interpretable features of the tasks.
b) The comparison between mean task-feature initialisation and the best- source oracle is useful, but I think an intermediate initialisation strategy would strengthen the conclusions. Given the clustering structure observed in the learned task-feature space, averaging the embeddings of all 99 pretrained tasks may obscure meaningful differences between task groups. It would be interesting to evaluate whether initialising from the mean feature vector of a cluster of related tasks improves adaptation compared with the global mean. More generally, could the learned similarity measures be used to predict good source initialisations without requiring the best-source oracle? This would help establish whether the learned task representations are useful for practical transfer, rather than only retrospectively explaining transfer performance.
c) I think the leave-one-out adaptation experiment would benefit from a more systematic evaluation of how generalisation depends on the number and diversity of pretrained tasks. Pretraining on 99 of the 100 tasks provides the model with exposure to almost the entire task battery, including numerous closely related tasks and compositional variants. Consequently, it is difficult to determine whether successful adaptation reflects the learning of broadly reusable computational structure or the availability of solutions to highly similar tasks during pretraining. It would be valuable to repeat this experiment with varying numbers of pretrained tasks and to evaluate how adaptation performance changes with the composition of the pretraining set. In particular, testing generalisation within and between task families could help establish whether the learned parameterisation supports transfer to genuinely different computational requirements. I think this would substantially strengthen the generalisation claims.
d) Finally, I think the discussion of catastrophic forgetting could be clarified. The authors correctly acknowledge that previously learned tasks are preserved by construction because the shared parameters are frozen during adaptation. While this is a useful property of the architecture, it does not directly demonstrate resistance to forgetting when shared parameters are allowed to change. I would therefore suggest distinguishing more clearly between avoiding forgetting through parameter freezing and mitigating interference during continued learning. Alternatively, an interesting extension would be to allow some shared parameters to adapt to new tasks and assess whether performance on previously learned tasks is retained.
Minor concerns:
These concerns largely cover clarity and terminology.
N.i) In 4.2 (Mechanisms for task separation), its slightly unclear which of the task separation methods belongs to which model, would the authors be able to provide a bit more detail on this?
N.ii) With respect to lines 098/099 - the authors say “Indeed, where tasks are structurally homogeneous, sharing parameters across tasks typically helps (Raffel et al., 2020).” - please clarify what sharing parameters helps with.
N.iii) Lines 168-169 “Each task k → {1, . . . , K} is assigned an embedding p(k) → RNfeat , with Nfeat ↔ dim(𝜃DSR)”, could you define DSR please? Or use the already-defined DS? Also, if my understanding is correct, the dynamical system you are parameterising is an RNN, if so, it may be in the interest of clarity to state this outright rather than saying as an example AL-RNN, unless you use other architectures.
N.iv) For figure 1A caption “Every task receives low-dimensional task features that modulate group-level parameters, producing task-conditioned AL-RNNs within a shared recurrent backbone.”, its slightly unclear immediately what you mean. The tasks dont receive the embeddings? I just think this is a brief wording issue.
N.v) An interesting additional analysis would be to explore the effective dimensionality of the learned task-parameter subspace. In Figure 2A, the hierarchical AL-RNN achieves approximately 95% accuracy with only Nfeat=4, approaching the performance achieved with 16-dimensional task features. This raises the question of whether all 16 available dimensions are effectively used. For example, the authors could construct a matrix containing the learned parameter vectors for all tasks and examine its singular value spectrum to estimate the effective dimensionality occupied by the task solutions. It would be interesting to determine whether the additional capacity is used to represent meaningful variation between tasks, or whether the learned solutions occupy a substantially lower-dimensional subspace.
N.vi) I have a slight concern about whether the flat architecture has fully converged by the end of training. While the hierarchical architecture clearly outperforms the flat model under the reported training conditions, it is less clear whether this difference reflects representational limitations or slower optimisation in the flat architecture. Figure 2B illustrates differences in task- specific learning times, but does not establish whether performance in the flat model has plateaued. Providing loss or accuracy curves, or extending training for the flat architecture, could help distinguish between these possibilities.
N.vii) It would be interesting to examine whether networks trained on the same target task converge to similar dynamical solutions when initialised randomly versus from a pretrained source task. For example, do successful transfers recover similar dynamical motifs to those found when training the target task from scratch? Conversely, does negative transfer lead to qualitatively different or less effective dynamical solutions? This would be really nice to see.
N.viii) In discussion lines 469-471, there is existing research on this which, interestingly, somewhat shows that sometimes representational similarity does not always facilitate learning. See, for example, Menghi et al. (Nature Communications, 2025). Here, they show that representational overlap between tasks may slow learning through increased interference. It would be interesting for you to discuss when shared computational structures might facilitate vs hinder efficient learning.
6 comments
Response to Reviewer 3Zvi (I)
Response to Weakness W.i) and M.i) a)–c):
We thank the referee for this important point, and agree that use of the term "shared backbone" was ambiguous in the current form. In line with a comment of Referee 1 (qwQq), we have now clarified why the interpretation of the shared task mechanisms in the ring attractor persists, even if the recurrence matrix can adjust across tasks.
Secondly, we took this comment as inspiration to test selective variants of the hierarchization, in which only a subset of the parameters is generated from the task features and the rest is shared across tasks. We have added the results to Sec. 4.1 and 4.3, and Appendix B.2. In short, sharing the input map, the readout, or the bias, input and readout together stays within seed variability of the full model (95.6–95.9% vs. 95.9%). Sharing the recurrent matrices , and across all tasks, with task-specific and , still reaches 93.4%, compared with 81% for the flat model. The losses concentrate on tasks requiring strongly nonlinear memory dynamics (i.e. the +int+seq variants, where all eight lose 7.6–17.8 points), which suggests that these benefit most from task-specific recurrence. We also checked how consistent the learned task organisation is across these variants. The variant with task-specific and reproduces the full model's organisation across seeds in all three similarity spaces (–, close to the full model's own seed-to-seed reproducibility), whereas the shared-recurrence variant departs more ( as low as in the task-feature space), suggesting that task organisation is carried mainly by the task-specific recurrent parameters (Appendix B.2).
Response to Weakness M.ii):
Our motivation for varying nonlinearity was threefold. (i) Prior work (Khona et al., 2023; Brenner & Koppe, 2026) suggests that many tasks can be solved by simple models, with linear or sparsely nonlinear dynamics, and we find this to be an important angle to track with increasing complexity of task batteries. The AL-RNN lets us vary the number of nonlinear units independently of state dimension, so we can ask how much nonlinearity a larger, more heterogeneous task suite actually requires. We find that a large subset of tasks is solvable fully linearly and that sparse nonlinearity already reaches good performance, and view this finding as an important confirmatory contribution, particularly because, in our view, (ii) it matters for biophysically inspired models, which are by their nature often highly nonlinear, and, vice versa, for interpreting neural recordings of task performance. (iii) Relatedly, a common intuition is that the nonlinearity of brain dynamics facilitates parameter sharing across tasks, which we wanted to explicitly investigate in our experiments, where several individually linear tasks cannot be solved jointly without additional nonlinear or hierarchical structure. Whether more nonlinearity then helps automatically in a large, heterogeneous battery was not clear a priori, and the flat model suggests it does not if the model is not further structured.
Following your suggestion, we added a new figure (Fig. 11 in the updated manuscript) showing accuracy as a function of for both architectures. In the flat model, mean accuracy peaks at sparse nonlinearity () and then declines, while the variation across seeds grows substantially. The best seeds stay high up to , so much of the decline reflects less reliable optimisation, not a lower performance ceiling per se, although for the best seeds also drop. In contrast, the hierarchical model improves up to – and then saturates, with very small variance across seeds. Together with the selective hierarchization results (see M.i), this suggests that sharing is broadly beneficial but that interference depends on task complexity: input and readout maps can be shared without loss, and shared recurrent matrices solve most tasks, whereas tasks requiring complex nonlinear dynamics deteriorate without task-specific recurrence. We flag the challenge of separating optimization difficulty from task interference and from the separation mechanism used, and flag as an open question for future work which kind of sharing is minimally required and most efficient, and how this interacts with task complexity.
References:
[1] Khona, M., Chandra, S., Ma, J. J., & Fiete, I. R. (2023). Winning the lottery with neural connectivity constraints: Faster learning across cognitive tasks with spatially constrained sparse RNNs. Neural Computation, 35(11), 1850-1869.
[2] Brenner, M., & Koppe, G. (2026). Uncovering the computational roles of nonlinearity in sequence modeling using almost-linear RNNs. Transactions on Machine Learning Research.
Response to Reviewer 3Zvi (II)
Response to Weakness M.iii) a)–c):
We thank the reviewer for these comments. We have revised Appendix A.2 in the updated manuscript, which now describes shared settings first and then each experiment separately. (a) Balancing holds per epoch: each task contributes a fixed pool of 200 trials, which is shuffled and split into batches of 64 every epoch for joint models, while minibatches themselves are not stratified. (b) All joint and individual models use five seeds. The leave-one-out experiment uses three pretraining seeds per held-out task. The details per experiment are updated in A.2. (c) We count trainable parameters only: for the hierarchical AL-RNN, the hypernetwork plus the task-feature vectors (formula in A.2), with all baselines matched to about 80k parameters.
Response to Reviewer 3Zvi
We are very thankful for this very thorough and constructive review! The many helpful suggestions prompted many new experiments and revisions, including ablations of which parameter groups are task-specific, a continual-learning experiment with trainable shared parameters, a new training-free selector for initialisations, and runs varying the number of pretrained tasks, among several clarifications in the manuscript. Some of these training runs are still finishing, and we are updating the manuscript accordingly. In order to ensure we can properly double check the new results, we will post our full point-by-point response together with the revised manuscript by Monday, November 16.