acceptodds
Under review as a conference paper at ICLR 2027

Maintain Your Ranks: Enabling Continual Learning in RL Agents with Growing and Pruning

Abstract

The learning capacity of neural networks in deep reinforcement learning (RL) is essential for extracting task-specific representations, thereby facilitating learning. However, networks' capacity can be lost during training with standard gradient optimizers; timely intervention is needed to provide new capacity or release the underutilized capacity to cope with the non-stationary data distribution in RL. Despite the evolving data distribution, common RL practices overly commit to static network architectures, which can suffer from plasticity loss in the sense of lacking adaptability in learning, an inherent trait that can be captured by their collapsing effective rank (e-rank). We argue that maintaining a sufficient e-rank of the neural networks' layers is a necessary precondition for plasticity. To mitigate plasticity loss, we suggest a dynamic adaptation of the architecture, aiming to maintain the e-rank, by growing and pruning the neurons of a layer. We demonstrate that RL agents with adaptively growing-and-pruning critics can continually learn in both standard continuous control RL and continual learning for sequential RL tasks with superior performance.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning, Continual learning, Adaptive architecture

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fTAMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes an effective-rank-guided dynamic architecture for mitigating plasticity loss in reinforcement learning. The method monitors the layer-wise effective rank of centered post-activations in SAC critics. When the effective-rank occupancy stops increasing, the corresponding layer is expanded; redundant neurons are subsequently pruned based on activation similarity. The method is evaluated on standard continuous-control tasks, random-friction environments, and sequential continual-learning tasks.

Critical strengths
  • The main idea is simple and intuitive: using effective rank not only as a diagnostic of plasticity loss, but also as an online signal for architecture adaptation.
  • The separation between the growth trigger and the growth-stopping criterion is sensible.
  • The experiments cover several forms of non-stationarity, including high-UTD training, changing dynamics, and sequential tasks.
  • The paper includes useful ablations and discusses a concrete failure case rather than hiding it.
Critical weaknesses
  1. The interpretation of the growth trigger is not fully justified.
    Growth is triggered when the occupancy, e-rank / width, plateaus. However, a plateau does not necessarily imply insufficient capacity; it may also indicate optimization convergence or that the current task simply does not require more feature dimensions. The stopping criterion partially addresses this issue, but only after extra capacity has already been added.

  2. The method appears sensitive to controller hyperparameters.
    The growth behavior depends on several quantities such as the measurement interval, window size, growth threshold, and stopping threshold. The failure on quadruped-run at high UTD is directly attributed to a miscalibrated stopping threshold. This raises concerns about how robustly these hyperparameters transfer across tasks and training regimes.

  3. The contribution of the e-rank mechanism is somewhat entangled with other optimization choices.
    The full method also uses LayerNorm, weight decay, cosine learning-rate scheduling, and learning-rate resets after growth. The ablations are helpful, but some of these components have substantial effects. I would like to see a cleaner comparison isolating the benefit of e-rank-guided adaptation over equally tuned schedule-based growth.

  4. The current scope is relatively narrow.
    The method is only demonstrated on fully connected SAC critics, while the actor remains fixed. It is therefore unclear how well the approach generalizes to other RL algorithms or architectures.

Questions for authors
  1. How sensitive are the results to the growth and stopping thresholds across environments and UTD ratios?
  2. Why is occupancy slope preferred over the slope of the absolute effective rank as the growth signal?
  3. How much wall-clock overhead is introduced by repeated effective-rank computation, clustering, and network resizing?
  4. Does the advantage remain if a schedule-based growing baseline is given the same learning-rate reset and optimization recipe?
Reviewer_a9mEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an effective-rank-guided grow-and-prune mechanism for SAC critics. Network width increases when representation utilization plateaus, while highly correlated or inactive neurons are pruned. The method is evaluated on standard continuous control, environments with changing friction, and sequential tasks. The strongest gains appear in the sequential dog tasks, whereas improvements over larger static critics are mixed. The main unresolved question is whether the mechanism offers benefits beyond strengthening the underlying network and training configuration.

Critical strengths
  • Relevant research problem. Maintaining adaptability under changing data distributions is important for continual RL. Using representation statistics to guide architectural changes provides a concrete alternative to fixed intervention schedules.

  • Meaningful treatment of redundancy. The pruning mechanism considers correlations between active neurons rather than relying solely on dormancy, addressing a form of redundancy that activation-magnitude criteria can overlook.

  • Useful empirical analysis. The paper includes static-width comparisons and ablations of growth triggers, pruning, initialization, and optimization settings. The sequential dog experiments provide evidence of improved adaptation over the evaluated baselines.

Critical weaknesses
  • The benefits may depend on limitations of the chosen SAC configuration. All experiments use SAC with relatively simple MLP critics. Figure 17 already shows that larger static networks can match or outperform the proposed method in several settings, with a substantial gap on quadruped-run at UTD 16. These results leave open whether dynamic adaptation is necessary or mainly compensates for insufficient capacity. Evaluation should include stronger backbones and algorithms such as FlashSAC and WarpSAC, comparing each original algorithm against its grow-and-prune variant under the same nonstationary protocols.

  • The interpretation of effective rank is stronger than the evidence supports. A plateau in effective-rank occupancy does not establish that network capacity is exhausted; it may also reflect optimization difficulties, limited data coverage, or sufficient task-specific representations. The observed association between effective rank and return does not establish that maintaining high effective rank is a necessary condition for plasticity. Additional controlled comparisons are needed to validate the proposed trigger.

  • The continual-learning evidence has a restricted scope. Task boundaries are known, replay buffers are cleared, and the growth-stopping criterion is reset at transitions. The curricula also progress through largely compatible skills, such as standing, walking, and running. These settings provide limited evidence for adaptation without explicit boundaries or retention under conflicting objectives. The permanent growth-stopping rule is particularly concerning when distribution shifts occur without a task-boundary reset.

  • The practical efficiency advantage is not established. Parameter savings alone do not demonstrate computational savings. Repeated SVD, similarity computation, clustering, and structural updates introduce overhead that should be measured. Comparisons with larger static critics should report both sample efficiency and wall-clock performance to establish whether the additional mechanism offers a favorable performance–cost trade-off.

Questions for authors

See Weakness

Reviewer_rx63OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a grow-and-prune method to mitigate plasticity loss in reinforcement learning. It periodically measures layer-wise effective rank to determine when to expand the critic’s hidden-layer widths, followed by activation-similarity-based pruning to remove redundant neurons. Experiments on standard continuous control, changing environment dynamics, and sequential tasks demonstrate competitive performance against existing methods, with improved adaptation in several settings. Ablations on the growth trigger, pruning, and optimization components, together with comparisons against fixed-width networks, provide further evidence of the method’s effectiveness and parameter efficiency.

Critical strengths
  • The method combines established ideas into a practical approach that achieves competitive performance and improved adaptation in several RL settings.
  • The evaluation covers multiple sources of non-stationarity, including high update-to-data ratios, changing environment dynamics, and sequential tasks.
  • The experiments examine both task performance and network size, showing that pruning can reduce parameter count while largely preserving performance.
Critical weaknesses
  • The methodological novelty appears incremental, with the main contribution lying in the specific integration of effective-rank-guided growth and activation-similarity-based pruning, rather than a fundamentally new approach to mitigating plasticity loss.
  • While the paper includes a periodic-growth comparison and component ablations, these do not fully isolate the benefits of the proposed selection criteria. Comparisons with alternative adaptive growth signals and pruning criteria within the same framework would help clarify their advantages. The main component ablations are also limited to Hopper-hop at UTD 1, leaving their generality across settings unclear.
Questions for authors
  • Does eRank-based growth outperform alternative triggers, such as dormant neuron ratios or gradient statistics, within the same framework?
  • Does activation-similarity pruning outperform magnitude- or dormancy-based pruning under the same growth conditions and pruning ratio?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.