acceptodds
Under review as a conference paper at ICLR 2027

Multitask Interactions in Recurrent Dynamics across 100 Cognitive Tasks

Abstract

The brain continually learns tasks with very different computational demands. Recurrent neural networks (RNNs) have been used as dynamical surrogates for studying task solutions, with recent work extending this approach to joint training on multitask batteries. However, it remains unclear how tasks interact as these batteries grow in scale and heterogeneity: under what conditions do they transfer or interfere with one another, and how do these effects relate to task complexity and model structure? We study this question by simultaneously training hierarchical Almost-Linear RNNs (AL-RNNs) on 100 tasks spanning a wide range of cognitive demands. AL-RNNs allow us to vary the amount of nonlinearity in the recurrence, while hierarchical task features provide a low-dimensional interface for task-specific adaptation within shared dynamics. On this task battery, performance improves with both nonlinear capacity and task-feature dimensionality, with the joint hierarchical model outperforming both individually trained models and a single unstructured joint model. Many tasks are individually solvable with linear dynamics, yet cannot be solved together by a shared linear model. Through hierarchical features, task-specific nonlinear usage, and task geometry, the trained model provides several related but distinct measures of task similarity that capture how computations are separated and combined within the shared backbone. Across a series of experiments, we relate these representations to individual task complexity, joint solvability, pairwise transfer learning, and leave-one-out adaptation. Together, our experiments characterize task interactions in multitask recurrent learning while demonstrating both the potential and the challenges of reusable dynamical backbones for cognitive computation.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Multitask learning, Recurrent neural networks, Cognitive task modeling, Transfer learning, Dynamical systems

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_3ZviOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The authors seek to better understand multitask learning in recurrent neural networks. They employ an impressively large task array (100 cognitive tasks), and use variations of an almost-linear RNN architecture (AL-RNN) to investigate the effects of hierarchical structure, nonlinearity, and task feature embedding on multitask learning. The paper makes several interesting contributions, including demonstrating that a hierarchical architecture can successfully solve a large battery of tasks despite generating task-specific RNN parameters from a low-dimensional embedding through a shared linear hyper network. This architecture maintains task-specific flexibility while constraining recurrent parameters to a common subspace, and outperforms flat jointly trained and individually trained networks, as well as LSTM, GRU, and vanilla RNN controls. The authors demonstrate that jointly solving tasks, even those which are individually linearly solvable, requires additional nonlinear capacity, highlighting potential mechanisms by which recurrent networks are able to do multitask learning. Furthermore, the paper employs three complementary methods of task similarity (learned task embeddings, piecewise linear activation region usage, and recurrent state geometry) to look at how task representations and dynamics organise. They relate these metrics to learning transferability, demonstrating that tasks which are more isolated within the similarity spaces generally provide less effective initialisations for learning other tasks. This paper provides evidence that structured low-dimensional parameter sharing can improve recurrent multitask learning, and offers interesting insights into task embeddings and transferability.

Critical strengths

S.i) Scale and scope is impressive - this paper studies multitask recurrent learning using an unusually large battery of 100 cognitive tasks, extending previous work that typically uses smaller and more structured task sets. This is a really cool addition to the literature base, and the authors conduct very nice examinations of how representational similarity influences task learning transfer.

S.ii) Strong architectural controls. This paper employs the AL-RNN in a hierarchical, flat, and individual manner, but also makes control comparisons with other recurrent architectures such as LSTM, GRU, and vanilla RNNs. The findings are made stronger by this.

S.iii) Interesting findings. I found the results particularly impressive and well structured. In particular, the relationship between individual task complexity and joint solvability, the range of similarity measures and investigation of these both across seeds and architectures, and the authors approach to ensuring robustness in results were appreciated. Furthermore, the hierarchical parameter sharing results are very interesting - that very few task-feature dimensions are sufficient to approach ceiling-level performance is suggestive that the hierarchical AL-RNN architecture can support diverse tasks through a very low-dimensional subspace is very impressive.

Critical weaknesses

W.i) The interpretation of sharing in the hierarchical architecture needs greater clarification and precision. Given that each task receives a distinct set of recurrent parameters, although these are generated from the same low- dimensional subspace, some claims concerning shared dynamics and shared recurrent backbone appear stronger than what is supported by the architecture.

Questions for authors

I found the paper really interesting and am broadly positive about its contributions. My main concerns relate to the interpretation of shared recurrent dynamics and transfer/generalisability results. I believe that the paper is largely very sound, so these concerns are suggestions to strengthen it.

Major concerns:

M.i) My primary conceptual concern relates to the interpretation of what is shared within the hierarchical architecture.

a) Lines 125-126 state that the framework should allow new tasks to be learned “without fitting an independent network for each task or modifying the shared dynamics.” This distinction should be clarified further. As I understand, the hierarchical AL-RNN architecture generates a separate set of recurrent parameters for each tsk, with these parameters generated from the shared low- dimensional subspace from G. There is clearly meaningful parameter sharing and reduced task-specific degrees of freedom, but each task is solved by a differently parameterised recurrent network. It would be good for the authors to clarify what they mean by avoiding independent networks or relation to shared dynamics.

b) More generally, terms such as “shared recurrent backbone”, “shared recurrent dynamics”, and “shared recurrent model” appear throughout the paper, and are potentially misleading when referring to the hierarchical AL- RNNs. This model produces task-specific networks, which do not share recurrent weights or dynamics, what is shared is the parameter generation subspace. While this does provide an important form of sharing, the authors must make clear the distinction between their model sharing recurrent parameterisation subspace and sharing recurrent dynamics, particularly when interpreting the mechanisms of task separation and reuse.

c) The clarity should be improved when discussing comparisons across tasks for the hierarchical AL-RNNs. Since the hierarchical architecture generates different recurrent parameters for each task, it is not immediately clear how geometric similarity between task-specific state spaces should be interpreted. Although the recurrent units have corresponding indices across tasks, their functional roles may differ. Could the authors clarify how this affects the interpretation, and whether shared parameterisation provides sufficient alignment for comparisons? Or alter these analyses to place task-specific networks into a fully shared subspace for comparisons? In particular, it would be helpful to distinguish similarity in state-space geometry from evidence for shared computational mechanisms. Overall, the paper would benefit a lot from distinguishing better between sharing a recurrent parameterisation subspace and sharing a recurrent substrate, particularly when interpreting the results.

M.ii) What is the relevance of varying the level of nonlinearity? I saw more of the motivation with regards to independently varying nonlinear capacity and latent dimensionality, but I think the primary motivation for using the AL-RNN is unclear, and the results concerning the effects of nonlinearity across architectures would benefit from further interpretation. The observation that performance in the flat model peaks with sparse nonlinearity vs hierarchical model benefits from full nonlinearity is interesting, but not discussed in much detail or interpreted. Could the authors elaborate on why additional nonlinearity appears detrimental to the flat architecture but beneficial for the hierarchical architecture? It’s especially interesting considering the idea that the flat model must encode all of the tasks in different regions of the recurrent space and switch between them using nonlinearity, but then seems to impair performance when there is more nonlinearity. It would be particularly interesting to understand whether this reflects differences in optimisation difficulty, task interference, or the dynamical mechanisms used to separate tasks.

M.iii) I found the training protocols difficult to follow from the main text and appendix A.2, I think further clarification is necessary to improve reproducibility. In particular:

a) Trials are sampled in batches of 64 with balanced task representation across the training pool, but it’s unclear whether individual minibatches are task balanced or whether tasks are sampled randomly with balancing occurring across the full training procedure.

b) Please clarify the number of random seeds used across architectures and analyses, and whether this varies between experiments.

c) Please provide detail on how parameter budgets are calculated and matched across architectures, particularly for the hierarchical AL-RNNs (i.e. it’s unclear whether the counted parameters are from G and p only - and how they are calculated).

M.iv) These points largely relate to the transfer and leave-one-out experiments, which I found to be very interesting, but believe that they lack some critical further investigation and controls which would strengthen the results.

a) The results relating task isolation to transferability are interesting, but I think they would benefit from further behavioural or computational interpretation. Approximately 30% of source–target pairs exhibit negative transfer relative to random initialisation. Are there identifiable task characteristics associated with these pairs, or with tasks that consistently act as poor transfer sources? Similarly, it would be useful to identify which tasks are most isolated in the similarity spaces shown in Figure 11, and whether their isolation corresponds to particular computational requirements. Finally, are tasks that perform poorly as initialisations in the pairwise transfer experiments also poor sources for leave- one-out adaptation? Exploring these relationships could help connect the learned similarity measures to more interpretable features of the tasks.

b) The comparison between mean task-feature initialisation and the best- source oracle is useful, but I think an intermediate initialisation strategy would strengthen the conclusions. Given the clustering structure observed in the learned task-feature space, averaging the embeddings of all 99 pretrained tasks may obscure meaningful differences between task groups. It would be interesting to evaluate whether initialising from the mean feature vector of a cluster of related tasks improves adaptation compared with the global mean. More generally, could the learned similarity measures be used to predict good source initialisations without requiring the best-source oracle? This would help establish whether the learned task representations are useful for practical transfer, rather than only retrospectively explaining transfer performance.

c) I think the leave-one-out adaptation experiment would benefit from a more systematic evaluation of how generalisation depends on the number and diversity of pretrained tasks. Pretraining on 99 of the 100 tasks provides the model with exposure to almost the entire task battery, including numerous closely related tasks and compositional variants. Consequently, it is difficult to determine whether successful adaptation reflects the learning of broadly reusable computational structure or the availability of solutions to highly similar tasks during pretraining. It would be valuable to repeat this experiment with varying numbers of pretrained tasks and to evaluate how adaptation performance changes with the composition of the pretraining set. In particular, testing generalisation within and between task families could help establish whether the learned parameterisation supports transfer to genuinely different computational requirements. I think this would substantially strengthen the generalisation claims.

d) Finally, I think the discussion of catastrophic forgetting could be clarified. The authors correctly acknowledge that previously learned tasks are preserved by construction because the shared parameters are frozen during adaptation. While this is a useful property of the architecture, it does not directly demonstrate resistance to forgetting when shared parameters are allowed to change. I would therefore suggest distinguishing more clearly between avoiding forgetting through parameter freezing and mitigating interference during continued learning. Alternatively, an interesting extension would be to allow some shared parameters to adapt to new tasks and assess whether performance on previously learned tasks is retained.

Minor concerns:

These concerns largely cover clarity and terminology.

N.i) In 4.2 (Mechanisms for task separation), its slightly unclear which of the task separation methods belongs to which model, would the authors be able to provide a bit more detail on this?

N.ii) With respect to lines 098/099 - the authors say “Indeed, where tasks are structurally homogeneous, sharing parameters across tasks typically helps (Raffel et al., 2020).” - please clarify what sharing parameters helps with.

N.iii) Lines 168-169 “Each task k → {1, . . . , K} is assigned an embedding p(k) → RNfeat , with Nfeat ↔ dim(𝜃DSR)”, could you define DSR please? Or use the already-defined DS? Also, if my understanding is correct, the dynamical system you are parameterising is an RNN, if so, it may be in the interest of clarity to state this outright rather than saying as an example AL-RNN, unless you use other architectures.

N.iv) For figure 1A caption “Every task receives low-dimensional task features that modulate group-level parameters, producing task-conditioned AL-RNNs within a shared recurrent backbone.”, its slightly unclear immediately what you mean. The tasks dont receive the embeddings? I just think this is a brief wording issue.

N.v) An interesting additional analysis would be to explore the effective dimensionality of the learned task-parameter subspace. In Figure 2A, the hierarchical AL-RNN achieves approximately 95% accuracy with only Nfeat=4, approaching the performance achieved with 16-dimensional task features. This raises the question of whether all 16 available dimensions are effectively used. For example, the authors could construct a matrix containing the learned parameter vectors for all tasks and examine its singular value spectrum to estimate the effective dimensionality occupied by the task solutions. It would be interesting to determine whether the additional capacity is used to represent meaningful variation between tasks, or whether the learned solutions occupy a substantially lower-dimensional subspace.

N.vi) I have a slight concern about whether the flat architecture has fully converged by the end of training. While the hierarchical architecture clearly outperforms the flat model under the reported training conditions, it is less clear whether this difference reflects representational limitations or slower optimisation in the flat architecture. Figure 2B illustrates differences in task- specific learning times, but does not establish whether performance in the flat model has plateaued. Providing loss or accuracy curves, or extending training for the flat architecture, could help distinguish between these possibilities.

N.vii) It would be interesting to examine whether networks trained on the same target task converge to similar dynamical solutions when initialised randomly versus from a pretrained source task. For example, do successful transfers recover similar dynamical motifs to those found when training the target task from scratch? Conversely, does negative transfer lead to qualitatively different or less effective dynamical solutions? This would be really nice to see.

N.viii) In discussion lines 469-471, there is existing research on this which, interestingly, somewhat shows that sometimes representational similarity does not always facilitate learning. See, for example, Menghi et al. (Nature Communications, 2025). Here, they show that representational overlap between tasks may slow learning through increased interference. It would be interesting for you to discuss when shared computational structures might facilitate vs hinder efficient learning.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 3Zvi

We are very thankful for this very thorough and constructive review! The many helpful suggestions prompted many new experiments and revisions, including ablations of which parameter groups are task-specific, a continual-learning experiment with trainable shared parameters, a new training-free selector for initialisations, and runs varying the number of pretrained tasks, among several clarifications in the manuscript. Some of these training runs are still finishing, and we are updating the manuscript accordingly. In order to ensure we can properly double check the new results, we will post our full point-by-point response together with the revised manuscript by Monday, November 16.

AuthorsOpenReview ↗

Response to Reviewer 3Zvi (I)

Response to Weakness W.i) and M.i) a)–c):

We thank the referee for this important point, and agree that use of the term "shared backbone" was ambiguous in the current form. In line with a comment of Referee 1 (qwQq), we have now clarified why the interpretation of the shared task mechanisms in the ring attractor persists, even if the recurrence matrix can adjust across tasks.

Secondly, we took this comment as inspiration to test selective variants of the hierarchization, in which only a subset of the parameters is generated from the task features and the rest is shared across tasks. We have added the results to Sec. 4.1 and 4.3, and Appendix B.2. In short, sharing the input map, the readout, or the bias, input and readout together stays within seed variability of the full model (95.6–95.9% vs. 95.9%). Sharing the recurrent matrices A\boldsymbol{A}, W\boldsymbol{W} and h\boldsymbol{h} across all tasks, with task-specific C\boldsymbol{C} and B\boldsymbol{B}, still reaches 93.4%, compared with 81% for the flat model. The losses concentrate on tasks requiring strongly nonlinear memory dynamics (i.e. the +int+seq variants, where all eight lose 7.6–17.8 points), which suggests that these benefit most from task-specific recurrence. We also checked how consistent the learned task organisation is across these variants. The variant with task-specific A\boldsymbol{A} and W\boldsymbol{W} reproduces the full model's organisation across seeds in all three similarity spaces (r=0.70r = 0.70–0.760.76, close to the full model's own seed-to-seed reproducibility), whereas the shared-recurrence variant departs more (rr as low as 0.430.43 in the task-feature space), suggesting that task organisation is carried mainly by the task-specific recurrent parameters (Appendix B.2).

Response to Weakness M.ii):

Our motivation for varying nonlinearity was threefold. (i) Prior work (Khona et al., 2023; Brenner & Koppe, 2026) suggests that many tasks can be solved by simple models, with linear or sparsely nonlinear dynamics, and we find this to be an important angle to track with increasing complexity of task batteries. The AL-RNN lets us vary the number of nonlinear units PP independently of state dimension, so we can ask how much nonlinearity a larger, more heterogeneous task suite actually requires. We find that a large subset of tasks is solvable fully linearly and that sparse nonlinearity already reaches good performance, and view this finding as an important confirmatory contribution, particularly because, in our view, (ii) it matters for biophysically inspired models, which are by their nature often highly nonlinear, and, vice versa, for interpreting neural recordings of task performance. (iii) Relatedly, a common intuition is that the nonlinearity of brain dynamics facilitates parameter sharing across tasks, which we wanted to explicitly investigate in our experiments, where several individually linear tasks cannot be solved jointly without additional nonlinear or hierarchical structure. Whether more nonlinearity then helps automatically in a large, heterogeneous battery was not clear a priori, and the flat model suggests it does not if the model is not further structured.

Following your suggestion, we added a new figure (Fig. 11 in the updated manuscript) showing accuracy as a function of PP for both architectures. In the flat model, mean accuracy peaks at sparse nonlinearity (P=16P = 16) and then declines, while the variation across seeds grows substantially. The best seeds stay high up to P=64P = 64, so much of the decline reflects less reliable optimisation, not a lower performance ceiling per se, although for P≥128P \geq 128 the best seeds also drop. In contrast, the hierarchical model improves up to P≈16P \approx 16–3232 and then saturates, with very small variance across seeds. Together with the selective hierarchization results (see M.i), this suggests that sharing is broadly beneficial but that interference depends on task complexity: input and readout maps can be shared without loss, and shared recurrent matrices solve most tasks, whereas tasks requiring complex nonlinear dynamics deteriorate without task-specific recurrence. We flag the challenge of separating optimization difficulty from task interference and from the separation mechanism used, and flag as an open question for future work which kind of sharing is minimally required and most efficient, and how this interacts with task complexity.

References:

[1] Khona, M., Chandra, S., Ma, J. J., & Fiete, I. R. (2023). Winning the lottery with neural connectivity constraints: Faster learning across cognitive tasks with spatially constrained sparse RNNs. Neural Computation, 35(11), 1850-1869.

[2] Brenner, M., & Koppe, G. (2026). Uncovering the computational roles of nonlinearity in sequence modeling using almost-linear RNNs. Transactions on Machine Learning Research.

AuthorsOpenReview ↗

Response to Reviewer 3Zvi (II)

Response to Weakness M.iii) a)–c):

We thank the reviewer for these comments. We have revised Appendix A.2 in the updated manuscript, which now describes shared settings first and then each experiment separately. (a) Balancing holds per epoch: each task contributes a fixed pool of 200 trials, which is shuffled and split into batches of 64 every epoch for joint models, while minibatches themselves are not stratified. (b) All joint and individual models use five seeds. The leave-one-out experiment uses three pretraining seeds per held-out task. The details per experiment are updated in A.2. (c) We count trainable parameters only: for the hierarchical AL-RNN, the hypernetwork GG plus the task-feature vectors p(k)\boldsymbol{p}^{(k)} (formula in A.2), with all baselines matched to about 80k parameters.

Reviewer_Pt5mOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies transfer and interference across a battery of 100 cognitive tasks using hierarchical Almost-Linear RNNs. Low-dimensional task features generate task-specific model parameters, while the number of nonlinear recurrent units can be varied independently. The analysis relates task performance and transfer to task-feature similarity, recurrent-state geometry, and linear-region usage. It also examines adaptation to held-out tasks by optimizing only their task features.

Critical strengths

The contribution is a systematic study of task interactions at a larger and more heterogeneous scale. The distinction between tasks being individually linearly solvable and jointly solvable is particularly interesting. Examining task features, state geometry, and nonlinear-region usage together provides a richer account of sharing and interference than performance comparisons alone.

Critical weaknesses

The comparison with individually trained models does not fully establish how much of the performance gain comes from sharing across tasks. It allows up to 3,000 training epochs for the joint models, compared with 500 for the individual models, with different warm-up and stopping settings. These differences do not necessarily make the comparison unfair, but evidence that the individual models have converged is needed to support the interpretation of the gap in Table 1.

Questions for authors

Have the individually trained models converged within the 500-epoch limit? Does the performance gap in Table 1 persist when they receive a longer training budget or a comparable learning-rate schedule?

1 comment
AuthorsOpenReview ↗

Response to Reviewer Pt5m

We would like to thank the reviewer for their time and feedback, and for a question that prompted an important clarification in the manuscript.

The apparent difference in schedule was mainly based on the different precise meaning of the term “epoch” in the individual vs. shared models, where the selected number of epochs corresponded to a comparable number of gradient updates in both model classes. With our training settings, an individual epoch has 7 updates on a single task, while an epoch in the joint model has 313 updates shared across 100 tasks. A 500-epoch cap thus gives each individual model up to 3,500 task-specific updates, comparable to the on 1,600–4,900 per task (mean: 3300 updates) the joint models used on average.

We nevertheless re-tested this analysis by providing a substantially larger training budget for the individual models (patience 300 epochs and a budget of 3000 epochs, (M=23, P=0 and P=16, 100 tasks × 5 seeds). For the nonlinear models, the more extensive runs were slightly better, but within 1% of the accuracy after 500 epochs. For the linear models, we found that a number of runs still continued improving after the 500-epoch cap, and accuracy improved from 65% to 70%. This value closely matches the jointly trained linear AL-RNN (70.7%), so the linear gap between joint and individual training disappears, while also indicating that for the individual linear models, a larger training budget is required for convergence.

We have updated Table 1 using the longer training protocol, and updated the description. We want to however emphasize that the main concern of the referee, i.e. the qualitative gap reported in the paper for the difference between hierarchical and individual (4-5%), and between linear and nonlinear models (25%), still holds with this setting.

Reviewer_qwQqOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies transfer and interference across 100 cognitive tasks using hierarchical Almost-Linear RNNs. Low-dimensional task embeddings generate task-specific network parameters, while nonlinear capacity can be varied independently of state dimension. The authors compare hierarchical, flat joint, and individually trained models under matched total parameter budgets, and characterize task relationships through embeddings, linear-region usage, and state-space geometry. Validation includes capacity sweeps, pairwise transfer, and adaptation to held-out tasks with shared parameters frozen. Hierarchical training improves overall performance, but transfer depends on the source task and initialization. Adapting only 16 task features reaches 84.5% accuracy from mean initialization and 92.1% with an oracle choice of starting task.

Critical strengths

The capacity study varies recurrent nonlinearity separately from state dimension and task-feature dimensionality. Across 100 tasks, introducing 2 nonlinear units raises hierarchical-model accuracy from roughly 70% to 89%, while 4 task features suffice to approach 95% with sufficient nonlinearity (4.1).

The representation analyses are connected to actual transfer experiments. Pairwise transfer and leave-one-out adaptation relate task features, linear-region usage, and state-space geometry to the reuse of learned solutions. The distinction between 92.1% adaptation accuracy with oracle initialization and 84.5% from mean initialization means that the shared model can support strong performance on new tasks, but finding those solutions remains difficult.

Critical weaknesses

Interpretation of region overlap as evidence of shared computation needs stronger support. Eq 3 makes the recurrent matrices task-dependent, so two tasks visiting the same activation region need not implement the same local dynamics. So, the lower task–region mutual information in the hierarchical model (4.2) establishes overlap in activation patterns, but does not establish greater reuse of recurrent computations. Similar ring-shaped trajectories in Fig 4 are suggestive, although the detailed stability analysis in Fig 5 examines only HeadDirection. Comparing the task-conditioned local dynamics across these examples would help distinguish shared dynamical mechanisms from different task-specific systems producing similar trajectories. This distinction matters because explaining how computations are reused is a central contribution beyond the performance benchmark.

Questions for authors

1 For the 3 ring-like task examples in Fig 4, how do the local dynamics compare within their overlapping activation regions—for example, their slow directions and stability? Since Eq 3 makes the recurrent matrices task-dependent, shared activation patterns need not implement shared computations.

  1. Eq 2 passes the first M-P units through linearly, but 3.1 defines their entries in the local activation matrix as zero. Should these entries instead be one, giving the local recurrent matrix (A+W\operatorname{diag}(\mathbf{1}_{M-P},b))? Were Fig 5’s fixed-point and stability calculations performed with this full matrix?

  2. Is the test loss used to calculate transfer scores in Eq 5 the raw masked MSE or the initialization-normalized loss from Eq 4? If it is normalized, is the denominator held fixed across source initializations?

1 comment
AuthorsOpenReview ↗

Response to Reviewer qwQq

We would like to thank the reviewer for their overall very positive assessment of our work. The three questions all raise valuable points, which we have addressed in the revised manuscript.

Q1 (Shared activation patterns vs. shared computations): This is an important point, and we agree that this framing was slightly misleading, since the hierarchization indeed means that the activity in the individual regions does not need to fully match. Our heuristic analysis rested on the point that i) the three tasks were very close in hierarchical feature space, meaning their recurrent dynamics only weakly changed across different tasks, ii) the assignment of the individual subregions still matched one-to-one across the tasks in terms of their location and local dynamics.

To make this precise, we have now analysed the local dynamics across the three tasks. First, we find that the regions consistently implement slow time scales, in line with the expectation of the slow manifold of the ring attractor. The per-region leading eigenvalue magnitudes are correlated between tasks (median r = 0.93–0.98). The slowest directions of the local Jacobians also geometrically align, pointing along nearly the same direction of state space. The main change is that the time constants are slightly adjusted, in line with the expectation of the task demands leading to subtly different requirements: HeadDirection must integrate a time varying angular velocity over 20-34 steps, while the +int tasks rotate at a constant velocity over a 8-17 step delay.

We have added and clarified this discussion within the manuscript (see new Section B.1 for a detailed description).

Q2 (Local activation matrix): Thank you for spotting this. There was indeed a typo in the model description in Section 3.1. The ReLU is applied only to the last P units, while the first M-P units are linear. The corresponding entries of D(bt)D(\mathbf{b}_t) are therefore 1, not 0, i.e. D~(bt)=diag⁡(1M−P,bt)\tilde{D}(\mathbf{b}_t) = \operatorname{diag}(\mathbf{1}_{M-P}, \mathbf{b}_t). We double checked all analysis code and confirmed that this typo is in text only, and all results in the paper are correct. We have updated the corresponding section in the manuscript.

Q3 (Transfer score loss): We thank the reviewer for pointing this out; the description was indeed ambiguous. The test loss in Eq. 5 is the unweighted masked MSE, i.e. the inner term of Eq.4 without the 1/Lk(0)1/L_k^{(0)} factor, averaged over the target task's test trials. No initialization-based normalization is applied, so there is no source-dependent denominator. The individual and transfer models were also trained with this unweighted loss; the Lk(0)L_k^{(0)} normalization is used only for joint multitask training, where it balances tasks with different target scales. Because the scores are standardized within each target (z-scored across the 99 sources), any fixed per-target rescaling of the loss would leave z(A→B)z(A \to B) and source quality unchanged. We have clarified this in Sec. A.2 of the revised manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General reply, summary of changes

We thank the referees again for their time and thoughtful feedback, and we are glad that all three shared an overall positive assessment of our work. The comments helped us better contextualize our results, and to sharpen some descriptions of our approach. Below we summarise the changes, with details in the individual responses.

New experiments and analyses

  • Selective hierarchisation (Sec. 4.1 and 4.3, App. B.2): model variants in which only some parameter groups are task-specific and the rest are shared, with their effect on performance and task organisation.
  • Local dynamics (App. B.1): comparison of the task-specific Jacobians, stability and slow modes of the three ring-attractor tasks within shared linear regions (Fig. 4A).
  • Transfer and adaptation (Sec. 4.4, App. A.3): zero-shot source selection for leave-one-out adaptation, adaptation after pretraining on subsets of tasks, and continual learning with plastic shared parameters, with and without replay.
  • Convergence and nonlinearity: training and validation loss curves (Fig. 5), accuracy as a function of nonlinearity for the flat and hierarchical models (Fig. 11), and individual models retrained with the same training budget (Table 1).

Clarifications and text edits

  • Appendix A: the training protocol is described per experiment (batching, seeds, loss used in each setting, parameter counting).
  • Terminology: throughout the paper, we now distinguish sharing a recurrent parameterisation subspace from sharing recurrent weights or dynamics.

Discussion

  • Expanded discussion of the transfer–interference trade-off and of which kinds of sharing are needed, as directions for future work, and integrated suggested literature.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.