acceptodds
Under review as a conference paper at ICLR 2027

Neuron Sharing in Multi-Task Reinforcement Learning Policies Follows the Geometry of the Task Parameterization

Abstract

Multi-task RL policies are usually treated as monolithic, yet prior work shows subnetworks for individual behaviors exist within them. We investigate how these subnetworks are structured within the network to identify neuron reusage and specificity. Using gradient-based importance profiles and causal ablation on policies trained with Proximal Policy Optimization, we measure neuron sharing between tasks. In a continuously parameterized locomotion task (AntDir), neuron sharing decays monotonically with angular distance between task goal directions, consistently across seeds and network widths. Removing one task's top neurons preferentially degrades its angular neighbors. This organization persists when all continuous goal features are replaced by one-hot task IDs, so it is learned from the behavioral geometry of the task family rather than inherited from the input encoding. In HalfCheetahVel, overlap also tracks velocity distance, but ablation damage is nonselective, showing that geometry-aligned overlap does not imply functional modularity. On a fully learned seven-task Meta-World subset, which lacks a continuous task parameterization, the ordering of pairwise overlap barely replicates across seeds and ablation effects are asymmetric. Together, these results indicate that neuron sharing in these policies follows the geometry of the task parameterization when one exists, and that overlap alone does not establish functional modularity.

Primary area:
reinforcement learning
Keywords:
multi-task reinforcement learning, mechanistic interpretability, neuron importance attribution, modularity, task geometry

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_9dBPOpenReview ↗
Rating
1
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This empirical paper asks how neurons in shared multi-task PPO policies are reused across tasks and whether the pattern reflects the geometry of task parameterizations. It computes task-conditioned neuron-importance profiles using gradient-times-activation at the penultimate layer, compares profiles through the Bhattacharyya coefficient, and performs task-targeted top-neuron ablations to test functional selectivity. Experiments use AntDir (eight goal directions), HalfCheetahVel (eight target speeds) and a seven-task learned subset of Meta-World MT10, with multiple seeds, two network widths where applicable, random-network controls, one-hot goal encodings and additional layer/pruning analyses. In AntDir, overlap and collateral ablation damage decrease with angular task distance, including under one-hot encoding. In HalfCheetahVel, overlap follows velocity distance but ablation damage is not target-selective. Meta-World shows overlap without robust task-pair ordering across seeds and exhibits asymmetric ablation effects. The paper appropriately distinguishes importance overlap from functional modularity.

Critical strengths
  1. A clear, controlled test of a precise representation question. The directional AntDir result is quantitatively strong: at width 64 under continuous encoding, mean overlap falls from 0.882 at 45° to 0.590 at 180° (Table 1), and an angular gradient remains when the continuous goal input is replaced by a one-hot task ID with comparable trained return (Sec. 4.1 and Appendix B). Random-initialization controls and first-layer analyses (Appendix F) help separate learned organization from trivial input similarity. Five seeds and width comparisons improve the credibility of the descriptive finding.
  2. Functional interventions and explicit negative cases prevent an overly simple modularity story. Targeted neuron ablations in AntDir preferentially damage nearby tasks (Table 2), while HalfCheetahVel shows little analogous selectivity despite overlap structure (Table 3). Meta-World shows asymmetric transfer of ablation damage, which symmetric overlap measures cannot capture (Table 5). The importance-guided pruning experiment (Appendix E) adds a limited but concrete use of the attribution profiles. This separation of similarity, causal vulnerability and directionality is informative.
Critical weaknesses
  1. An unresolved attribution-versus-state-distribution confound weakens the mechanistic interpretation. For the chosen penultimate layer with a linear action head, the Jacobian is the same readout-weight matrix for every task; the reported gradient-times-activation profiles therefore largely reweight the task-dependent magnitudes of hidden activations by fixed output weights (Sec. 3.1). Profiles are further averaged over separately collected rollouts for each task. Differences in visited states, action magnitude or occupancy may consequently induce the observed geometric structure without showing that reusable task-specific circuitry is organized by geometry. Causal neuron ablations provide useful support, but they do not by themselves identify which source creates the attribution pattern.
  2. The scope of robust, geometry-aligned functional specialization is narrow. A selective angular ablation gradient is established most clearly in AntDir, a small and highly structured eight-task family. In HalfCheetahVel, the overlap gradient is dissociated from selective functional damage (Sec. 4.2), while in the seven-task Meta-World subset the pairwise ordering reproduces only weakly between seeds (mean Spearman 0.21/0.30 at widths 64/400; Table 4). Three of ten MT10 tasks are excluded from the main analysis because the policies did not learn them (Sec. 4.3; Appendix H). These limitations reduce the case for a general claim about multi-task policies beyond this descriptive set of PPO MLPs. Figure/presentation concern (secondary): Figures 1 and 3 place many small numerical annotations inside heatmap cells and use dense rotated task labels; the multiple colored activation traces and bands in Fig. 2 are also difficult to distinguish at normal paper size. Because pairwise comparisons are central to the paper, the figures should be redesigned with larger labels, reduced in-cell clutter and more distinguishable encodings. This is not by itself evidence that the underlying measurements are wrong.
Questions for authors
  1. Matched-state attribution (assessment-critical). If all task importance profiles are recomputed on a shared pool of observations (or with a state-distribution-matched importance estimator), rather than on task-specific rollouts, does the AntDir angular overlap curve survive? A related activation-only baseline would help determine how much of the gradient-times-activation result is attributable to learned task-dependent computation versus occupancy and activation scale. This would materially affect the mechanistic interpretation.
  2. Generality and causal selectivity (assessment-critical). Can the authors provide a task-normalized measure of ablation selectivity that adjusts for baseline return and the magnitude of disruption, especially in HalfCheetahVel, where returns can become negative? Is there an additional continuous-goal task family with nontrivial state variation in which the AntDir pattern replicates? Either result would materially change the assessment of how broadly the proposed organizing principle holds.
Reviewer_C9VVOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to understand how neurons are shared between tasks in multi-task reinforcement learning (MTRL). In order to experiment with different MTRL settings, the authors use ant environment with different velocities, HalfCheetah with different velocities, and Meta-World as their problem settings. The submission uses the strength of a gradient and it's activation as their main metric for comparisons, and use the Bhattacharyya Coefficient to compare gradient x activation scores. The submission finds that different velocities seem to induce less sharing the further the velocity is compared to a reference velocity. Similar results are found for the different tasks in Meta-World.

Critical strengths
  1. I think it is a very common assumption in MTRL that neural networks share structure between tasks, this paper seems like an interesting first step in understanding that idea.

  2. The results seem to hold, at this stage, across multiple environments so the finding seems to generalize well past traditional MTRL benchmarks like Meta-World.

Critical weaknesses
  1. I really like the ideas explored here, but it does seem like this is only the first step of a scientific paper. The submission has identified a phenomenon, that neurons are shared between related tasks, but there is no next step. There is no remedy, there is no exploration of the phenomenon under different MTRL conditions (some ideas on this later in this paragraph), etc. This could be trying to make dissimilar tasks more similar somehow, or investigating different how this effect changes across different network sizes (e.g. combine the current submission with the work of McLean et al, 2025b (the scaling paper), across different architectures (e.g. vanilla networks vs BRC https://arxiv.org/pdf/2505.23150), or even tasks of different state/action spaces (e.g. I think BRC has experiments on Meta-World and DMC in a single network, from https://arxiv.org/abs/2310.16828). Perhaps even further analysis of why the 7 tasks of MT10 are learned but not the removed ones could be useful.

I think this work has the potential to be a really important finding for MTRL, but I don't think it currently goes far enough. I think after reading the paper and thinking that the findings are interesting, I'm left thinking "yes, and?" To be clear, I really do like the problem that the authors are interested in. I just think the authors can go even further than their current results.

  1. I have quite a few issues with the writing of this paper. I think there are many improvements that can be made to the presentation of this paper that would also strengthen the submission. The introduction dives right into MTRL, without any context. There is no mathematical formulation of the MTRL problem (for some of the paper I was a bit confused on whether the paper was solving multi-task supervised learning or MTRL). Another example is in the presentation of the results. On line 248 the authors say "Figure 1 shows a clear geometry-respecting structure in AntDir." but this is said without any context, any description of what experiment is being run/examined, etc. This leaves the reader to have to discern what is going on by themselves. I think the authors should provide more guidance into the results with this in mind.

  2. Perhaps related to the mathematical formulation portion, I think a visual depiction of certain things could be super useful. For example, I'm not really sure what "AntDir policies with all continuous goal features replaced by an 8-dimensional one-hot vector" means. But a visual explanation couldd be really useful here.

Questions for authors

Future work question. In the Future work section, the authors mention "task-conditioned routing." Isn't this the network architecture that McLean et al, 2025b argue against?

Reviewer_uVsKOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper looks at how neurons in a multi-task RL policy are shared between tasks. The authors score how much each neuron in the last hidden layer matters for each task, compare these scores across tasks, and then knock out each task's top neurons to see which tasks break. In AntDir, tasks with nearby directions share more neurons and hurt each other more when ablated, and this holds even when the goal is given as a one-hot task ID. In HalfCheetahVel the overlap follows velocity but ablation hurts all velocities about equally, and in Meta-World there is no stable structure. The takeaway is that neuron sharing follows the geometry of the task family when there is one, and that overlap alone doesn't mean modularity.

Critical strengths
  • The question is clear and the paper is easy to read. I like that the training setup is kept fixed and only the task family, goal encoding and width are changed.
  • The one-hot experiment is a nice control. It shows the angular structure in AntDir isn't simply coming from the goal vector in the input.
  • The AntDir vs HalfCheetahVel contrast is probably the most useful part of the paper. Both give similar looking overlap matrices but behave very differently under ablation, which is a good warning for anyone reading modularity off similarity plots.
  • Results are over 5 seeds and several sparsity levels, and the authors are upfront about negative results, e.g. PPO not learning 3 of the MT10 tasks.
Critical weaknesses
  • I don't find the main findings to be very surprising or new. Variations of the main result, that related tasks reuse overlapping sets of neurons or subnetworks according to task similarity, have already been shown in prior multi-task and RL subnetwork studies (see the missing references below, as well as Driscoll et al. 2024 which the paper cites), and the point that similarity heatmaps don't prove causal modularity is already a standard reason people run ablations in interpretability.

  • The experimental scope is also small. Everything is 2-layer MLPs trained with PPO, and the positive results come from two simple locomotion tasks controlled by a single parameter. I'd want to see the analysis on deeper networks, other RL algorithms, or harder multi-task benchmarks before calling this a general organizing principle.

  • Because the output head is linear, the importance score at the last hidden layer is just the activation scaled by the outgoing weights. The authors acknowledge this, but it means the argument for using gradients over plain activations doesn't really add anything for the main analysis.

  • The one-hot control doesn't fully remove geometry from the input. A trained policy walking north sees different states (e.g. its own velocity) than one walking south, while the untrained baseline barely moves. So part of the angular structure could come from the states visited rather than from the network learning to route tasks.

  • I'm not convinced by the HalfCheetahVel ablation result. At width 64 the top 5% is only 3 neurons, and the same few neurons seem to rank highest for every velocity, so each ablation may be removing nearly the same neurons. In that case uniform damage is expected and doesn't say much about modularity. Ranking neurons by how task-specific they are, rather than by raw importance, would be a fairer test.

  • The width comparison is confounded. The 400-unit AntDir policies get much lower return than the 64-unit ones (this only shows up in the appendix), so the claims about wider networks specializing more compare a good policy with a weaker one. HalfCheetahVel is also never run at width 400.

  • The Meta-World asymmetry story leans heavily on drawer-close, which barely degrades under any ablation, including its own. If removing its "top" neurons doesn't hurt it, those probably aren't the neurons it actually relies on. The error bars on these numbers are also very large for 5 seeds.

  • The related work misses several close papers that should be in the background/literature review:

    • Yang et al., Task representations in neural networks trained to perform many cognitive tasks, Nature Neuroscience 2019. They cluster individual units by how much each task uses them and find groups that follow task similarity, which is very close to the question asked here.
    • Arnob et al., Efficient Reinforcement Learning by Discovering Neural Pathways, NeurIPS 2024. They find sparse task-specific pathways inside one shared network for multi-task continuous control, which relates directly to the paper's opening claim about subnetworks and to its pruning experiment.
    • Vischer et al., On Lottery Tickets and Minimal Task Representations in Deep Reinforcement Learning, ICLR 2022. They show that sparse masks in RL policies reveal which parts of the network a task actually needs, which is relevant to using importance scores to find subnetworks.
    • Yang et al., Multi-Task Reinforcement Learning with Soft Modularization, NeurIPS 2020, and Sun et al., PaCo: Parameter-Compositional Multi-Task Reinforcement Learning, NeurIPS 2022. Both build explicit task-specific routing or composition on Meta-World, so they are the natural comparison for asking whether modularity emerges in a plain MLP.
Questions for authors
  • How much do the top-5% masks for different velocities overlap in HalfCheetahVel?
  • Do the width results still hold if the 400-unit model is tuned to reach similar return?
  • If importance is computed on the same fixed set of states for every one-hot task, is the angular structure still there?
  • In Meta-World, does computing importance only over the part of the episode before the task is solved change the drawer-close result?

Minor:

  • Section 3.4 says 8 velocities from 0.25 to 1.75, but the tables go up to 2.00.
  • Several citations should be parenthetical.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.