acceptodds
Under review as a conference paper at ICLR 2027

AIPR:Adaptive Isometric Plasticity Regularization for Reinforcement Learning

Abstract

Deep reinforcement learning encounters a balance breakdown between stability and plasticity during long-term training, which causes a degradation in the network's continuous optimization capability and impairs the agent's long-term learning ability. Some existing approaches employ the Deviation from Isometry (DfI) to measure how far a weight matrix strays from an isometric structure. However, discrete parameter reinitialization interrupts the continuous optimization process, while abrupt parameter updates may disrupt the policy distribution. To address this issue, we propose Adaptive Isometric Plasticity Regularization (AIPR). This method transforms DfI from a trigger metric for discrete parameter reinitialization into a continuously differentiable geometric regularization term. By jointly optimizing this term alongside the reinforcement learning task objective, AIPR continuously constrains the evolving geometric state of network weights throughout training. Concurrently, AIPR incorporates a DfI-aware adaptive gating mechanism that dynamically regulates regularization intensity based on the network's current weight geometric state. Considering potential discrepancies in weight geometry between the Actor and Critic, AIPR separately characterizes their geometric states, generates corresponding adaptive regularization weights, and preserves the original task coupling within the Actor-Critic framework. AIPR is compatible with both on-policy and off-policy learning, enabling seamless integration with PPO and SAC. Empirical results demonstrate that AIPR achieves competitive learning performance across multiple continuous control benchmarks.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Actor-Critic, Plasticity Loss, Adaptive Regularization

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_wBLLOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Adaptive Isometric Plasticity Regularization (AIPR), which incorporates a continuously differentiable Deviation from Isometry (DfI) regularization term into actor-critic reinforcement learning. The method adaptively adjusts regularization strengths for the actor and critic and is evaluated with PPO and SAC on continuous control benchmarks.

Critical strengths

None that can meaningfully outweigh the fundamental concerns regarding the reliability and integrity of the scholarly references.

Critical weaknesses

n/a

Questions for authors

n/a

Reviewer_8SyzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Adaptive Isometric Plasticity Regularization (AIPR), a geometric regularization method for actor-critic reinforcement learning. It is based on deviation from isometry (DfI) and proposes a differentiable regularization objective. The proposed method also allows adaptive strength of the regularization through a sigmoid gating function (based on the current network geometry). The authors also present three theoretical properties of their method, which show that small DfI preserves well-conditioned weight spectra and non degenerate feature covariance under certain assumptions, while adaptive gating reduces an upper bound on gradient interference. The method is evaluated with PPO and SAC across different RL environments (Mujoco, Robosuite, CARL).

Critical strengths
  • The method is simple and compatible with existing methods in the actor-critic literature.
  • The paper uses several environments (mujoco, robosuite, carl) and two RL algorithms (PPO and SAC), as well as 10 random seeds.
  • The paper provides theoretical analysis of the geometric regularizer and adaptive gating mechanism. The derivations provide conditional guarantees rather than establishing preservation of RL plasticity.
Critical weaknesses
  • The continuous fixed-strength (Continuous+Fixed DfI, Table 1) ablation seems to perform poorly on several tasks (e.g. ~ -20 on Ant and ~20 on HalfCheetah), whereas the adaptive method achieves much higher returns. However, the way the fixed-strength parameter has been tuned is not described in the paper. Therefore, it cannot be established whether the performance gap is due to adaptive strength or due to a poorly tuned baseline. Since the adaptive coefficient is always smaller than its maximum, the differences could reflect the regularization strength rather than adaptivity.
  • Missing validation of the proposed mechanism: The paper proposes a method based on weight geometry and plasticity preservation but does not report singular value distributions, gate dynamics, or direct plasticity measurements (e.g. dead neurons, rank of the policy, policy variance, etc.)
  • Bibliography full of errors: The current reference list contains a majority of incorrect author names (e.g. Moalla et al. 2024, Han et al. 2026, Chung et al 2024 etc.). Given the number of incorrect author names, it raises concerns about the reliability of the paper's presentation, as well as on the reference verification.
Questions for authors

1 - What regularization coefficient was used for Continuous + Fixed DfI in Table 1? How does AIPR compare to a tuned fixed coefficient baseline? 2 - Can the authors provide evidence of plasticity preservation when using AIPR? 3 - How does AIPR compare against unregularized PPO and SAC?

Reviewer_DSP3OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Previous work has established that isometric weights are amenable to continual learning due to a better preservation of weight plasticity. However, these previous works (FIRE, ) achieved this structurally. This work proposes a 'soft' auxiliary objective with dynamic weighting.

Critical strengths
  • [S1] Replacing a hard structural constraint with an adaptive auxillary loss is a sensible idea worth investigating
Critical weaknesses
  • [W1] AI-generated citations. I have not done a full analysis of the references, but the 'Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning' citation in the references lists 'Mohammad Nauman' as the first author when the first author is Michal Nauman. I assume this is due to LLM hallucination. This is unacceptable and grounds for desk rejection in my opinion. The robosuite citation also lists Yuke Zhu twice. I also noticed that the Parseval paper lists William Chung as the first author when it's Wesley Chung! The authors also do not disclaim AI generated references and also claim they checked all LLM outputs. Based on the state of the references I cannot believe this is true.

  • [W2] No consideration of update-to-data ratio / UTD / replay ratio / RR. The FIRE paper considers the high UTD setting as this is where the effects of plasticity is encountered most strongly. There is no mention of this in the paper.

  • [W3] No fair reproduction of previous experimental results. FIRE had a specific experimental protocol that used DQN on Atari and Simba-v2 on HumanoidBench. The paper does not reproduce this experiment so it's hard to verify it outperforms FIRE on its own terms.

  • [W4] Missing baselines. The paper does not compare to or reference CrossQ+WN (Palenicek et al., NeurIPS 2025) or XQC (Palenicek et al., ICLR 2026) that achieve SOTA plasticity using batch norm and weight norm. Also the SIMBA papers (Lee et al. ICLR 2025, ICML 2025) which uses norms on the weights and activations.

  • [W5] Results nowhere near SOTA. Since the authors don't list the specific MuJoCo gym suite version it's hard to compare, but the results look nowhere near optimal based on the recent SOTA papers (see W4). Looking at the XQC performance curves (Appendix E.4) and listing rough numbers

EnvReturn@ Steps
Hopper-v43000250K
Ant-v47000500K
HalfCheetah-v410000250K
Walker2D-v45000250K
Humanoid-v410000250K

The results in Figure 3 are nowhere near this therefore we can't tell if their proposed plasticity regularization is useful for achieving best possible sample efficiency.

  • [W6] Lack of detail in related work. The baselines in this paper, FIRE, Parseval and PFO are not thoroughly described in the related work, which makes it hard to understand the experimental results. I see there is more information in the Appendix, but this is not ideal

Minor comments

W6. On line 86, \citep is used when \citet is required, i.e.

Dohare et al. (Dohare et al., 2024)

The title should have a space after the colon, i.e. 'AIPR: Adaptive Isometric...'

The text in Figure 1 has two typos: 'smail' and 'gemetry'

Questions for authors

None.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.