acceptodds
Under review as a conference paper at ICLR 2027

COME ON AGENT, LET’S STAY PLASTIC: WELL-CONDITIONED OPTIMISATION FOR MULTI- TASK AND META-RL

Abstract

Meta-learning with gradient adaptation assumes that pretraining on many tasks leaves a network that learns new tasks faster than a random one. We show that this can fail in deep reinforcement learning: networks pretrained on many tasks in parallel with Adam or Muon adapt to held-out tasks more slowly than a freshly initialised network. We ask whether optimisers that keep the weights close to orthogonal, which preserve plasticity when tasks arrive in sequence, also preserve adaptability after parallel pretraining. We study AdamO and MuonO, which add a decoupled soft orthogonality regulariser to Adam and Muon, and Parseval regularisation, and relate few-step adaptation to the neural tangent kernel of the pretrained network. In pre-registered experiments in two gridworld families, the regularised optimisers preserve the adaptability of deep networks and, at moderate width, adapt better than a fresh network, and they match or beat plasticity baselines and entropy controls. In the settings we tested outside gridworlds, pretraining did not reduce adaptability. The choice of pretraining optimiser therefore matters for gradient adaptation where pretraining reduces adaptability.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Plasticy, Isometry, Multi Task RL, Meta RL

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_C3gYOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the effects of leveraging neural network optimizers that are designed for maintaining orthogonality between tasks when the data arrives in sequence, on problems where there are multiple tasks. There is also an analysis of these optimizers when meta-learning is performed.

Critical strengths

None.

Critical weaknesses
  1. The writing of the paper is heavily influenced by AI. I'm really not sure what a "pre-registered study," an "arm", "primary contrasts," or "pre-registered with its primary contrast, with 64 fresh seeds" is. The paper becomes very difficult to parse and understand very quickly.

  2. The introduction of the paper immediately starts talking about meta-learning. Why do we use meta-learning? What is the goal of meta-learning? There is no description of what meta-learning is anywhere in this paper. There is no mathematical formulation of meta-learning, no mathematical formulation of reinforcement learning (there is a mention of an MDP but that is it). For so much of the paper, I had no idea what problem was actually being solved.

  3. It seems like the paper is built on an agent trained via PPO. But, PPO is not a meta-learning algorithm? MAML or RL^2 are meta-learning algorithms, shouldn't they be used to train the agent?

  4. The related works section reads just like a list of papers.

Questions for authors

See weaknesses.

Reviewer_bWcJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks whether a network pretrained in parallel on many RL tasks remains a good initialization for later gradient-based adaptation. In two gridworld settings, the authors find that deep PPO networks pretrained with Adam or Muon adapt to held-out tasks more slowly than a fresh network. AdamO, MuonO, and Parseval regularization reduce this loss. The paper also presents a linearized NTK analysis and reports diagnostics such as feature rank, Jacobian conditioning, policy entropy, and per-task kernel scales.

Critical strengths

The paper separates performance on pretraining tasks from usefulness as an initialization for later adaptation. That distinction matters for multi-task RL, fine-tuning, and MAML-style methods.

Critical weaknesses
  1. It seems that this paper's contribution is incremental. Compared with existing AdamO and Parseval, MuonO mainly adds the same decoupled orthogonality correction to Muon. The theory is also mostly a repackaging of standard ideas. Gradient descent in a linearized model is kernel spectral filtering; multi-task kernels split into within-task and cross-task blocks; near-orthogonal linear layers control singular-value growth. These results are useful, but they do not give new guarantees for PPO, Adam, or non-stationary RL training.
  2. The strong positive results come from NineRooms and a constructed Permuted DoorKey variant. The DoorKey experiment is especially limited: the authors tried several candidate settings and selected the first one where Adam passed the training criterion but still lagged behind a fresh network. The fixed order helps, but it still means the second positive environment was selected because it showed the target failure mode.
Questions for authors
  1. Under identical learning rates, do AdamO and MuonO still beat L2-init, entropy-tuned Adam, and Parseval?
  2. Why does strict entropy matching work for only 2 of 64 Adam seeds? Given this failure, how can you rule out entropy as the main mechanism?
Reviewer_vSS2OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper explores the impact of initial task training on downstream task adaptation, comparing between different optimizers and orthogonalizations. The paper's main point is that sometimes pretraining over an initial set of tasks hurts downstream adaptation, and attempts to profile the conditions where this can happen, focusing in particular on the optimizers. The paper compares several optimizers and regularizations, under two sets of task setups, a 15x15 gridworld and 5x5 DoorKey maze.

Critical strengths

This paper explores a potential failure mode of pretraining: after looking for setups where pretraining does not lead to downstream generalization, it then explores expanded setups to see where generalization does start to happen. These appear to be related to orthogonalization and expanded model size, even for the very small task setups studied. The fact that generalization might be improved by these, even for very small pretraining pools, is a small but still somewhat interesting clue.

Critical weaknesses

The setup is too small and not varied enough: The conclusions center around when pretraining hurts downstream adaptation, but in the main set of experiments in sec 6.1, pretraining is on only 20 total tasks. This is extremely small for a pretraining set. And downstream adaptation uses a total of 11 tasks (6 downstream and 5 val). And the seed seems it may vary only the model init and random sampling, so the 31 tasks here is 31 total for the entire 6.1 set of expers. I think this is far too small a testbed to draw any conclusions that generalize beyond this one selection of rooms. Sec 6.2 DoorKey is somewhat better and more varied, but still small. And many of the conclusions are based on the even smaller set in 6.1.

In addition, the fact that the pretraining pool is only 20 tasks suggests that the model may just have overfit to these tasks, before being adapted to the downstream ones. So the poor downstream performance may be due to initial overfitting --- not a failure with pretraining generally. But "pretraining" usually refers to training on large, diverse pools, with transfer enabled by the diversity. If the number of pretraining tasks were expanded to be larger, there may indeed be better transfer. So I think the claim of this being a pretraining failure case is an over-reach, if the cause was actually just overfitting.

The paper is also difficult to understand and follow. I asked one key point on clarification below to confirm my interpretation of the task setup. But had more trouble understanding throughout, including how to interpret the figures, which rely heavily on the coefficient of variation, but never explain why this metric is used or why it matters to the analysis.

Questions for authors
  • I had trouble understanding the parallel pretraining setup. Sec 5 says there are 20 goals in parallel. But presumably the episodes are sequential. And Sec B says 16 environments run in parallel, and "a goal" is drawn for each episode. What is the setup? Possibly this means one (different) goal is drawn for each of the 16 environments for each training episode, and all goals are drawn from the same training pool of 20 total?

  • "Our pre-registered prediction that the AdamO gain over Adam grows with depth failed" --- I didn't see this described as a hypothesis or question anywhere else in the paper, how is it pre-registered or even asked as a question before?

  • Same for sec 6.3 "Our pre-registered hypothesis, saturation of the deep tanh units, passes its test, but controls show that it is not the cause"

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.