- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes an effective-rank-guided dynamic architecture for mitigating plasticity loss in reinforcement learning. The method monitors the layer-wise effective rank of centered post-activations in SAC critics. When the effective-rank occupancy stops increasing, the corresponding layer is expanded; redundant neurons are subsequently pruned based on activation similarity. The method is evaluated on standard continuous-control tasks, random-friction environments, and sequential continual-learning tasks.
Critical strengths
- The main idea is simple and intuitive: using effective rank not only as a diagnostic of plasticity loss, but also as an online signal for architecture adaptation.
- The separation between the growth trigger and the growth-stopping criterion is sensible.
- The experiments cover several forms of non-stationarity, including high-UTD training, changing dynamics, and sequential tasks.
- The paper includes useful ablations and discusses a concrete failure case rather than hiding it.
Critical weaknesses
-
The interpretation of the growth trigger is not fully justified.
Growth is triggered when the occupancy,e-rank / width, plateaus. However, a plateau does not necessarily imply insufficient capacity; it may also indicate optimization convergence or that the current task simply does not require more feature dimensions. The stopping criterion partially addresses this issue, but only after extra capacity has already been added. -
The method appears sensitive to controller hyperparameters.
The growth behavior depends on several quantities such as the measurement interval, window size, growth threshold, and stopping threshold. The failure on quadruped-run at high UTD is directly attributed to a miscalibrated stopping threshold. This raises concerns about how robustly these hyperparameters transfer across tasks and training regimes. -
The contribution of the e-rank mechanism is somewhat entangled with other optimization choices.
The full method also uses LayerNorm, weight decay, cosine learning-rate scheduling, and learning-rate resets after growth. The ablations are helpful, but some of these components have substantial effects. I would like to see a cleaner comparison isolating the benefit of e-rank-guided adaptation over equally tuned schedule-based growth. -
The current scope is relatively narrow.
The method is only demonstrated on fully connected SAC critics, while the actor remains fixed. It is therefore unclear how well the approach generalizes to other RL algorithms or architectures.
Questions for authors
- How sensitive are the results to the growth and stopping thresholds across environments and UTD ratios?
- Why is occupancy slope preferred over the slope of the absolute effective rank as the growth signal?
- How much wall-clock overhead is introduced by repeated effective-rank computation, clustering, and network resizing?
- Does the advantage remain if a schedule-based growing baseline is given the same learning-rate reset and optimization recipe?