acceptodds
Under review as a conference paper at ICLR 2027

Decoupled Policy Adaptation for Non-Stationary Reinforcement Learning

Abstract

Catastrophic forgetting is a key obstacle for reinforcement learning (RL) in non-stationary environments, where dynamics may repeatedly alternate across related regimes, causing adaptation to the current context to overwrite behaviors needed under previously encountered ones. Many prior methods rely on task identifiers or restrictive update constraints, making them less suitable for continuously varying contexts or online sample-efficient adaptation. We propose Decoupled Adaptation of Policies via HyperNetwork Interactions (DAPHNI), a sample-efficient architecture for continuous context-driven non-stationary RL that mitigates forgetting through decoupled, similarity-driven adaptation. DAPHNI decomposes a Gaussian policy into a state-only base policy to preserve shared behavior and a context-dependent residual modulator to localize context-specific adaptation. To handle continuously varying contexts, the residual modulator is implemented by a state-conditioned hypernetwork that induces multiplicative state–context interactions. A Kullback-Leibler (KL) divergence constraint further anchors the final policy to the base policy. We theoretically show that DAPHNI yields similarity-driven updates, transferring adaptation across aligned contexts while suppressing interference on dissimilar ones to mitigate forgetting. Comparison experiments and ablation studies show consistent gains over representative baselines across multiple benchmarks and diverse context dynamics, while mechanism analysis supports our design.

Primary area:
reinforcement learning
Keywords:
non-stationary reinforcement learning, catastrophic forgetting, policy decomposition, hypernetworks, sample-efficient adaptation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JwyLOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DAPHNI, a policy architecture for mitigating catastrophic forgetting in RL with continuously varying contexts. The Gaussian policy is decomposed into a state-only base policy and a context-dependent residual produced by a state-conditioned hypernetwork, with a KL penalty anchoring the final policy to the base. Under an NTK approximation, the authors show that the residual update at one context is proportional to its inner product with the training context. Experiments on MuJoCo benchmarks and Lunar Lander report higher returns than several baselines.

Critical strengths

Strengths

  1. The paper addresses an important and underexplored problem: catastrophic forgetting in RL when the context changes continuously, without task boundaries.

  2. The proposed architecture is simple and lightweight. It adds only a base policy, a hypernetwork-based residual, and a KL term, so it can be easily attached to off-policy algorithms such as SAC and combined with context-inference methods (Appendix C).

  3. Proposition 1 offers an interpretable account of how the multiplicative state-context interaction shapes residual updates, linking the architectural choice to cross-context interference.

  4. The evaluation covers a broad range of non-stationary settings, including four MuJoCo environments with five context dynamics (smooth, noisy, abrupt, and multivariate) and Lunar Lander with simultaneous transition and reward changes.

  5. The ablation study examines each component (modulator, KL constraint, base policy, and interaction type), and the authors make an effort to probe the underlying mechanism through base policy extraction and an influence analysis.

  6. The KL coefficient is tuned automatically via a target-KL dual update, and the sensitivity analysis (Appendix B.2) shows that performance is fairly robust to the choice of target KL.

Critical weaknesses
  1. Insufficient evidence of forgetting mitigation. The only forgetting-specific metric, CRC (Tables 1 and 7), is reported only for DAPHNI and its ablations, not for any baseline, and only on Half Cheetah. Final returns (Figure 4, Table 4) cannot distinguish less forgetting from faster learning, and the smoothed, seed-averaged curves in Figure 3 (with independently sampled wind phases per seed) can hide drops when a previous context recurs. CRC itself is also questionable: it is positive for every method (+5.5% to +27.1%), so it conflates retention with continued learning, and its normalization by R_before favors methods with low initial performance. More direct evidence is needed, such as CRC for the baselines or periodic evaluation at fixed contexts during training.

  2. Fairness and coverage of the baselines. PT-DQN is applied via action discretization and MBCD without its model-based rollouts (Appendix D.2), so their poor performance may reflect these modifications rather than forgetting. The comparison also omits the alternatives closest to DAPHNI's components, such as a hypernetwork-based context-conditioned SAC or SAC regularized toward an information-restricted default policy.

  3. Ablations limited to Half Cheetah. All ablations and analyses (Tables 6-8, Figure 6) are conducted only on Half Cheetah. Since the paper claims larger gains in higher-dimensional environments such as Ant, ablations there are needed to show that the findings generalize.

  4. Weak analysis of the base-residual decomposition. The extracted base policy is compared only with a random policy (Tables 2 and 8), which is an uninformative reference; a state-only SAC policy would better test whether the base captures shared behavior. Moreover, the extracted base reaches only about 38-64% of the full DAPHNI return, implying a large residual, which conflicts with the premise in Section 4.1 that the context-specific component is small and localized.

Minor

  1. Evaluation metric. J_eval in Eq. 1 is a discounted return, but with gamma = 0.99 the reported values in the thousands (e.g., 4044.7 and 5076.9 on Half Cheetah) appear to be undiscounted sums. In addition, Section 3 states that online cumulative return is the primary metric, whereas the results are measured in separate evaluation episodes with a deterministic policy (Table 11, Figure 3, Appendix D.3). Please clarify how the reported metric is computed.

  2. Notation. The symbol t denotes the environment time step in Section 3, Eq. 1, and Algorithm 1, but denotes training (gradient-flow) time in Proposition 1 and Appendix A without being redefined. Similarly, T is the episode horizon in Eq. 1 but the training-time interval in Lemma 1. Using a separate symbol for training time would avoid confusion.

Questions for authors

Questions

  1. Relation to prior KL-regularized methods. Anchoring the policy to a state-only base policy via a KL penalty resembles prior approaches such as learned default policies with restricted inputs [1] and Distral [2]. How does DAPHNI differ from these methods?

  2. Missing baseline curves in Figure 3. Could the authors provide learning curves for all baselines, at least in the appendix? In particular, OGD, identified in Figure 7 as the strongest baseline under the discrete wind form (4413.4 in Table 4), is missing from the discrete panel of Figure 3.

  3. Context recurrence and training length. What are the periods of the wind schedules, and how often does each context region recur within 100K steps? Since several curves in Figure 3 are still rising, longer training would help clarify whether the gains reflect mitigated forgetting or faster early learning.

  4. Relation to hypernetwork-based context conditioning. Prior work [3] has explored both directions of combining state and context with hypernetworks: Hyper-MAML uses the context as the hypernetwork input, while its PEARL critic uses the state as the hypernetwork input with the context fed to the generated network. Could the authors discuss how the design in Section 4.2 relates to these prior choices?

[1] Information Asymmetry in KL-Regularized RL, ICLR 2019

[2] Distral: Robust Multitask Reinforcement Learning, NeurIPS 2017

[3] Recomposing the Reinforcement Learning Building Blocks with Hypernetworks, ICML 2021

Suggestions

  1. Clarifying Proposition 1. Please define z_A and z_B at their first appearance (page 2) and state that the proportionality to z_A^T z_B holds with s, s', and z_B fixed. It would also help to clarify what the "current context z_B" means when each replay batch mixes many contexts, and whether the inner product remains a meaningful similarity for the raw context coordinates used in the multivariate and Lunar Lander settings.

  2. Positioning with respect to meta-RL and contextual RL. The related work treats meta-RL mainly as a context-inference tool, although meta-RL and contextual RL are naturally formulated as CMDPs. Discussing related lines of work, such as continuous adaptation in nonstationary environments [4], zero-shot generalization across contexts [5], and hypernetwork policies conditioned on known context such as HyperZero [6], would clarify the contribution.

  3. Relation to shared-backbone adaptation. The motivation in Section 4.1, a shared policy component with lightweight context-specific adaptation, resonates with recent work such as APB [7]. The authors may find it useful to discuss how DAPHNI's residual relates to such shared-backbone formulations.

[4] Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments, ICLR 2018

[5] A Survey of Zero-Shot Generalisation in Deep Reinforcement Learning, JAIR 2023

[6] Hypernetworks for Zero-Shot Transfer in Reinforcement Learning, AAAI 2023

[7] Adaptive Policy Backbone via Shared Network, ICML 2026

12 comments

Thank you for your detailed review and concrete suggestions for improving the retention evidence, component analysis, and positioning of DAPHNI.

Weakness 1: Matched historical-context evaluations separate return losses from continued improvement.

We have replaced CRC with MNRC and expanded historical-context evaluation to every comparison baseline on all four MuJoCo benchmarks. The definition and results are reproduced below.

Mean Negative Return Change (MNRC) directly measures historical performance loss after adaptation elsewhere. For NN valid pairs, returns Rbefore,i>0R_{\mathrm{before},i}>0 and Rafter,iR_{\mathrm{after},i} are measured before and after intervening training with matched initializations and the same context trajectory:

MNRC=1N∑i=1Nmin⁡(0,Rafter,i−Rbefore,iRbefore,i).\mathrm{MNRC}=\frac{1}{N}\sum_{i=1}^{N}\min\left(0,\frac{R_{\mathrm{after},i}-R_{\mathrm{before},i}}{R_{\mathrm{before},i}}\right).

Matched evaluations isolate historical performance changes. Gains are clipped to zero before averaging, so they cannot offset losses; all valid pairs, including zeros, count. Changes of +20+20\\% and −10-10\\% give −5-5\\% MNRC. MNRC is nonpositive; closer to zero means stronger retention. We use deterministic evaluation from 10 fixed initial states and matched wind trajectories, excluding the first 30K steps from pair selection. We average pairs within each seed, then valid seed scores across 10 runs.

We evaluate MNRC only under the four wind dynamics because the paired protocol tests a historical high-wind region after training toward the opposite extreme; their ordered scalar context defines this comparison consistently. Multivariate contexts lack this unique ordering.

For a fixed absolute loss, a smaller positive reference return makes MNRC more negative; positive gains receive zero irrespective of the denominator. The 30K-step exclusion avoids unstable early reference policies. Scores are computed from paired evaluations within each seed, independently of learning-curve smoothing or cross-seed wind phases, and are read alongside absolute returns.

Mean MNRC (%); closer to zero is better. Bold marks the best mean.

Half Cheetah

\text{Method} & \text{Sine} & \text{Triangular} & \text{Random} & \text{Discrete} \\\\ \hline \text{DAPHNI} & \mathbf{-2.78} & -1.13 & \mathbf{-5.82} & \mathbf{-0.31} \\\\ \text{LCPO} & -3.01 & \mathbf{-1.02} & -9.19 & -0.34 \\\\ \text{SAC} & -12.73 & -8.66 & -16.71 & -13.92 \\\\ \text{OGD} & -3.14 & -3.60 & -8.39 & -2.51 \\\\ \text{MBCD} & -16.23 & -9.80 & -18.00 & -4.22 \\\\ \text{PT-DQN} & -12.80 & -11.27 & -15.57 & -6.87 \\\\ \end{array}$$ Hopper $$\begin{array}{lrrrr} \text{Method} & \text{Sine} & \text{Triangular} & \text{Random} & \text{Discrete} \\\\ \hline \text{DAPHNI} & -3.91 & -2.94 & \mathbf{-8.22} & -2.01 \\\\ \text{LCPO} & \mathbf{-3.48} & \mathbf{-2.81} & -10.26 & \mathbf{-1.65} \\\\ \text{SAC} & -10.86 & -8.60 & -12.60 & -9.16 \\\\ \text{OGD} & -5.90 & -5.53 & -10.35 & -6.33 \\\\ \text{MBCD} & -18.25 & -12.57 & -19.05 & -4.72 \\\\ \text{PT-DQN} & -14.96 & -13.19 & -18.88 & -8.48 \\\\ \end{array}$$ Walker2d $$\begin{array}{lrrrr} \text{Method} & \text{Sine} & \text{Triangular} & \text{Random} & \text{Discrete} \\\\ \hline \text{DAPHNI} & \mathbf{-4.14} & \mathbf{-2.19} & \mathbf{-7.54} & -2.27 \\\\ \text{LCPO} & -5.48 & -2.95 & -10.93 & \mathbf{-1.66} \\\\ \text{SAC} & -12.12 & -8.60 & -17.01 & -12.55 \\\\ \text{OGD} & -6.52 & -4.58 & -9.84 & -6.05 \\\\ \text{MBCD} & -15.75 & -10.30 & -19.07 & -5.64 \\\\ \text{PT-DQN} & -13.05 & -10.94 & -15.53 & -8.78 \\\\ \end{array}$$ Ant $$\begin{array}{lrrrr} \text{Method} & \text{Sine} & \text{Triangular} & \text{Random} & \text{Discrete} \\\\ \hline \text{DAPHNI} & \mathbf{-3.51} & -1.83 & \mathbf{-7.05} & \mathbf{-1.31} \\\\ \text{LCPO} & -4.39 & \mathbf{-1.49} & -8.92 & -1.76 \\\\ \text{SAC} & -15.45 & -10.95 & -16.86 & -9.02 \\\\ \text{OGD} & -4.78 & -3.43 & -9.14 & -4.83 \\\\ \text{MBCD} & -18.13 & -11.99 & -19.82 & -4.23 \\\\ \text{PT-DQN} & -14.73 & -13.21 & -15.80 & -7.15 \\\\ \end{array}$$ Notice that Online Cumulative Return remains our main metric, jointly reflecting forgetting resistance and online adaptation efficiency. DAPHNI achieves the highest mean return in all 20 MuJoCo settings. The MNRC results above directly assess historical retention, with DAPHNI ranking first in 10 of 16 wind settings and within the top two throughout. Together, these results demonstrate strong retention alongside efficient online adaptation. In the revised PDF, Section 3 defines MNRC and Appendix D.3 specifies the evaluation protocol. Section 5.2, Table 1 reports the results reproduced above, and Figures 3–4 provide the return results. Appendix D.4, Table 7 supplies the corresponding seed-level standard deviations.

Weakness 2: Baseline adaptations and component comparisons clarify the scope of the evidence.

PT-DQN retains permanent–transient value decomposition with branching action discretization, while MBCD retains probabilistic context detection and policy reuse and trains policies from real transitions. We make these continuous-action benchmark adaptations explicit, and assess retention additionally against continuous-action SAC, LCPO, and OGD. DAPHNI combines higher returns with strong MNRC against these baselines. For the closest components, we compare the hypernetwork modulator without a base, concat and inverted modulators, and w/o KL, and add independently trained state-only SAC and base extraction from w/o KL. These comparisons examine decomposition, interaction, and anchoring directly. The relationship to learned-default-policy methods is addressed in our response to Question 1.

Mean ablation MNRC (%) on Half Cheetah. Rows and columns are transposed for readability; bold entries mark the best mean.

\text{Method} & \text{Sine} & \text{Triangular} & \text{Random} & \text{Discrete} \\\\ \hline \text{DAPHNI} & -2.78 & \mathbf{-1.13} & -5.82 & \mathbf{-0.31} \\\\ \text{w/o modulator} & -12.73 & -8.66 & -16.71 & -13.92 \\\\ \text{w/o KL} & -3.09 & -5.28 & -14.05 & -3.76 \\\\ \text{concat modulator} & -3.52 & -1.71 & -12.58 & -8.27 \\\\ \text{w/o base policy} & \mathbf{-1.60} & -1.24 & \mathbf{-2.20} & -0.84 \\\\ \text{inverted modulator} & -6.88 & -3.45 & -8.57 & -4.19 \\\\ \end{array}$$ Removing the base confines updates to the similarity-driven modulator and removes a shared source of cross-context change, naturally explaining the better MNRC of w/o base policy in some settings. The modulator must then also learn shared behavior: under Random dynamics, return falls from $4604.1$ to $2202.3$ despite MNRC improving from $-5.82\\%$ to $-2.20\\%$. Retaining the base supports much stronger behavior with strong retention. In the revised PDF, Appendix D.2 specifies the baseline adaptations; Section 5.2 reports the comparison evidence in Figures 3–4 and Table 1. Section 5.3 gives the component ablations, including Table 2 reproduced above, and Section 5.4 provides the new extraction comparisons.

Weakness 3: Ant ablations and mechanism studies extend the component findings to quadruped control.

We extended both ablation and mechanism studies to Ant. DAPHNI achieves the highest mean return under all five Ant dynamic forms and the best ablation MNRC under Triangular, Random, and Discrete dynamics. Its extracted base outperforms the base from w/o KL in every Ant setting, and the interaction probe reproduces the concentration of update influence on matching and nearby contexts. The component findings therefore extend to higher-dimensional quadruped control.

Mean ablation returns on Ant. Bold entries mark the best mean.

\text{Method} & \text{Sine} & \text{Triangular} & \text{Random} & \text{Discrete} & \text{Multi} \\\\ \hline \text{DAPHNI} & \mathbf{1212.3} & \mathbf{958.5} & \mathbf{817.7} & \mathbf{1087.8} & \mathbf{1107.7} \\\\ \text{w/o modulator} & 724.2 & 710.9 & 523.2 & 836.4 & 708.2 \\\\ \text{w/o KL} & 1013.1 & 807.8 & 594.0 & 765.5 & 844.2 \\\\ \text{concat modulator} & 932.9 & 755.1 & 500.5 & 930.7 & 802.4 \\\\ \text{w/o base policy} & 850.2 & 754.1 & 399.0 & 737.6 & 621.0 \\\\ \text{inverted modulator} & 1045.3 & 823.8 & 672.5 & 905.0 & 922.9 \\\\ \end{array}$$ Mean ablation MNRC (%) on Ant; closer to zero is better. Bold entries mark the best mean. $$\begin{array}{lrrrr} \text{Method} & \text{Sine} & \text{Triangular} & \text{Random} & \text{Discrete} \\\\ \hline \text{DAPHNI} & -3.51 & \mathbf{-1.83} & \mathbf{-7.05} & \mathbf{-1.31} \\\\ \text{w/o modulator} & -15.45 & -10.95 & -16.86 & -9.02 \\\\ \text{w/o KL} & -4.95 & -7.38 & -16.72 & -5.14 \\\\ \text{concat modulator} & -6.39 & -4.72 & -17.33 & -6.79 \\\\ \text{w/o base policy} & \mathbf{-1.76} & -1.99 & -7.47 & -1.73 \\\\ \text{inverted modulator} & -8.47 & -3.48 & -9.70 & -4.32 \\\\ \end{array}$$ As in Half Cheetah, removing the shared branch confines updates to the similarity-driven modulator, naturally explaining the better Sine MNRC of w/o base policy. Its Sine return falls from $1212.3$ to $850.2$, while DAPHNI leads both return and MNRC under the other three wind dynamics. Mean base policy extraction returns on Ant; "base" denotes the extracted branch evaluated without further training. $$\begin{array}{lrrrrr} \text{Policy} & \text{Sine} & \text{Triangular} & \text{Random} & \text{Discrete} & \text{Multi} \\\\ \hline \text{DAPHNI base} & 756.5 & 563.5 & 412.2 & 624.0 & 454.6 \\\\ \text{DAPHNI full} & 1212.3 & 958.5 & 817.7 & 1087.8 & 1107.7 \\\\ \text{state-only SAC} & 670.5 & 518.4 & 457.4 & 723.2 & 492.5 \\\\ \text{w/o KL base} & 564.3 & 355.0 & 287.6 & 390.8 & 227.2 \\\\ \text{w/o KL full} & 1013.1 & 807.8 & 594.0 & 765.5 & 844.2 \\\\ \end{array}$$ In the revised PDF, Sections 5.3–5.4 point to the Ant results in Appendix D.4. The three tables reproduced above give the Ant results from Table 8 (ablation returns), Table 9 (ablation MNRC), and Table 10 (base extraction); Figure 11 shows the hypernetwork interaction probe. **Weakness 4: Extraction controls support shared behavior and the contribution of KL anchoring.** The new extraction comparison directly uses independently trained state-only SAC. DAPHNI's extracted base achieves 86–116% of its return without additional training, and exceeds the base extracted from w/o KL by 27–68%. The comparison also extends to Ant. This supports useful shared behavior and the role of KL in maintaining that behavior. The base/full return ratio measures the outcome of executing two policies, not the magnitude of their action differences: even one changed decision near a trap can determine whether a goal is reached. The local decomposition motivates the architecture without requiring globally small corrections. Full DAPHNI's additional return demonstrates the value of context-specific adaptation on top of the shared behavior. Mean evaluation returns on Half Cheetah. Rows and columns are transposed for readability; "base" denotes the extracted branch evaluated without further training. $$\begin{array}{lrrrrr} \text{Policy} & \text{Sine} & \text{Triangular} & \text{Random} & \text{Discrete} & \text{Multi} \\\\ \hline \text{DAPHNI base} & 2606.4 & 2491.2 & 2357.6 & 2830.4 & 1311.5 \\\\ \text{DAPHNI full} & 4044.7 & 4944.1 & 4604.1 & 5076.9 & 3452.8 \\\\ \text{state-only SAC} & 2247.9 & 2717.1 & 2327.0 & 3136.1 & 1523.4 \\\\ \text{w/o KL base} & 1856.5 & 1960.6 & 1579.1 & 2013.0 & 779.4 \\\\ \text{w/o KL full} & 3294.5 & 4323.3 & 3541.6 & 4056.6 & 2809.6 \\\\ \end{array}$$ In the revised PDF, Section 4.2 gives the local decomposition. Section 5.4, Table 3 reports the extraction results reproduced above, and Appendix D.4, Table 10 extends the comparison to Ant.
Reviewer_rKYpOpenReview ↗
Rating
2
Confidence
1(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates catastrophic forgetting in non-stationary reinforcement learning, where environmental contexts change continuously or abruptly over time. The authors propose DAPHNI, a decoupled policy architecture that separates a state-only base policy from a context-dependent residual modulator. The residual component uses a state-conditioned hypernetwork to generate context-specific policy adjustments, while a KL divergence constraint limits deviations from the base policy. The authors further provide a theoretical analysis based on the Neural Tangent Kernel (NTK) approximation, arguing that policy updates are influenced by contextual similarity.

The method is implemented within the Soft Actor-Critic (SAC) framework and evaluated on four MuJoCo benchmarks and Lunar Lander under different non-stationary dynamics. Comparisons with five baseline methods, component ablations, context-retention evaluations, and mechanism analyses are presented to support the claimed improvements in adaptation performance and forgetting mitigation.

Critical strengths
  1. The paper addresses catastrophic forgetting in reinforcement learning under continuously changing environmental contexts, without relying on predefined task boundaries or task identifiers. This is a practically relevant problem, as conventional continual learning methods often assume discrete task transitions.

  2. The authors evaluate DAPHNI across multiple MuJoCo environments and Lunar Lander under different non-stationary dynamics. The experiments include comparisons with representative baselines, component ablations, and additional analyses of context retention and policy behavior. The consistently higher average returns suggest potential practical benefits, although these results do not necessarily establish that the improvements originate from reduced catastrophic forgetting.

Critical weaknesses
  1. Although the paper claims to mitigate catastrophic forgetting, the experimental evidence does not convincingly distinguish improved learning performance from actual knowledge retention. Most results emphasize cumulative returns, which may reflect better optimization or increased model capacity rather than reduced forgetting. The proposed Context Retention Change (CRC) metric is also insufficient to establish this claim, as all reported CRC values are positive and may simply reflect continued learning. More importantly, removing the base policy achieves higher CRC values than the complete DAPHNI in three of four settings, which raises questions about the claimed role of the base policy in preserving previous knowledge. Without more direct forgetting evaluations against the main baselines, the paper's central claim remains inadequately supported.

  2. The theoretical analysis establishes a similarity-dependent relationship between residual policy updates under idealized NTK assumptions, but this does not necessarily imply reduced catastrophic forgetting. In particular, the derivation focuses on residual policy updates while the actual algorithm jointly updates both the base policy and the residual modulator. Since the base policy is continuously updated, the KL constraint between the final and base policies does not guarantee preservation of previously learned behaviors. Furthermore, contextual similarity measured through inner products does not necessarily correspond to similarity in optimal behaviors or the degree of interference between them. Consequently, the theoretical results do not sufficiently establish the claimed connection between similarity-driven updates and forgetting mitigation, leaving the central motivation of the proposed architecture insufficiently justified.

Questions for authors
  1. How can the authors demonstrate that DAPHNI genuinely mitigates catastrophic forgetting rather than simply improving overall learning performance? The reported CRC values are all positive, and the variant without the base policy achieves higher CRC in three of the four settings. Can the authors provide more direct forgetting evaluations, particularly comparisons of historical-context performance against the main baselines, to substantiate the central claim?

  2. How does the proposed architecture prevent previously learned behaviors from being overwritten when the base policy itself is continuously updated? Since both the base policy and residual modulator are jointly optimized, and the KL constraint only measures divergence from the current base policy, what mechanism ensures that the base policy retains previously acquired knowledge? Can the authors provide empirical evidence that the base policy maintains historical behaviors throughout training?

  3. To what extent does Proposition 1 actually explain the claimed reduction in catastrophic forgetting? The theoretical result establishes a relationship between contextual similarity and residual policy updates under NTK assumptions, but smaller cross-context updates do not necessarily imply less forgetting. Can the authors clarify how this relationship translates into improved retention of previously learned behaviors, particularly when the theoretical analysis does not account for simultaneous updates to the base policy?

3 comments

Thank you for your careful review and suggestions for strengthening the evidence on historical-context retention and the roles of DAPHNI's components.

Weakness 1: MNRC directly measures historical performance loss across all comparison baselines.

We have added direct historical-context evaluations against all comparison baselines on all four MuJoCo benchmarks and replaced CRC with MNRC. The definition and results are reproduced here.

Mean Negative Return Change (MNRC) is defined as follows: "Mean Negative Return Change (MNRC) directly measures the loss of previously acquired performance after adaptation to other contexts." For each of NN valid historical-context evaluation pairs, Rbefore,i>0R_{\mathrm{before},i}>0 and Rafter,iR_{\mathrm{after},i} are measured with matched initializations and the same context trajectory, before and after intervening training:

MNRC=1N∑i=1Nmin⁡(0,Rafter,i−Rbefore,iRbefore,i).\mathrm{MNRC}=\frac{1}{N}\sum_{i=1}^{N}\min\left(0,\frac{R_{\mathrm{after},i}-R_{\mathrm{before},i}}{R_{\mathrm{before},i}}\right).

Matching evaluation conditions isolates changes in historical performance. Clipping each pair before averaging makes gains contribute zero, so they cannot cancel losses; all valid pairs, including zeros, remain in the average. For example, changes of +20+20\\% and −10-10\\% give MNRC of −5-5\\%. MNRC is nonpositive, and values closer to zero indicate stronger forgetting resistance. The protocol uses deterministic evaluation from 10 fixed initial states, matched wind trajectories, within-seed averaging, and exclusion of the first 30K training steps from pair selection. The reported score averages valid seed-level scores across the 10 runs.

We evaluate MNRC only under the four wind dynamics because the paired protocol tests a historical high-wind region after training toward the opposite extreme; their ordered scalar context defines this comparison consistently. Multivariate contexts lack this unique ordering.

Mean MNRC (%; closer to zero is better), split by benchmark for readability. Bold entries mark the best mean.

Half Cheetah

\text{Method} & \text{Sine} & \text{Triangular} & \text{Random} & \text{Discrete} \\\\ \hline \text{DAPHNI} & \mathbf{-2.78} & -1.13 & \mathbf{-5.82} & \mathbf{-0.31} \\\\ \text{LCPO} & -3.01 & \mathbf{-1.02} & -9.19 & -0.34 \\\\ \text{SAC} & -12.73 & -8.66 & -16.71 & -13.92 \\\\ \text{OGD} & -3.14 & -3.60 & -8.39 & -2.51 \\\\ \text{MBCD} & -16.23 & -9.80 & -18.00 & -4.22 \\\\ \text{PT-DQN} & -12.80 & -11.27 & -15.57 & -6.87 \\\\ \end{array}$$ Hopper $$\begin{array}{lrrrr} \text{Method} & \text{Sine} & \text{Triangular} & \text{Random} & \text{Discrete} \\\\ \hline \text{DAPHNI} & -3.91 & -2.94 & \mathbf{-8.22} & -2.01 \\\\ \text{LCPO} & \mathbf{-3.48} & \mathbf{-2.81} & -10.26 & \mathbf{-1.65} \\\\ \text{SAC} & -10.86 & -8.60 & -12.60 & -9.16 \\\\ \text{OGD} & -5.90 & -5.53 & -10.35 & -6.33 \\\\ \text{MBCD} & -18.25 & -12.57 & -19.05 & -4.72 \\\\ \text{PT-DQN} & -14.96 & -13.19 & -18.88 & -8.48 \\\\ \end{array}$$ Walker2d $$\begin{array}{lrrrr} \text{Method} & \text{Sine} & \text{Triangular} & \text{Random} & \text{Discrete} \\\\ \hline \text{DAPHNI} & \mathbf{-4.14} & \mathbf{-2.19} & \mathbf{-7.54} & -2.27 \\\\ \text{LCPO} & -5.48 & -2.95 & -10.93 & \mathbf{-1.66} \\\\ \text{SAC} & -12.12 & -8.60 & -17.01 & -12.55 \\\\ \text{OGD} & -6.52 & -4.58 & -9.84 & -6.05 \\\\ \text{MBCD} & -15.75 & -10.30 & -19.07 & -5.64 \\\\ \text{PT-DQN} & -13.05 & -10.94 & -15.53 & -8.78 \\\\ \end{array}$$ Ant $$\begin{array}{lrrrr} \text{Method} & \text{Sine} & \text{Triangular} & \text{Random} & \text{Discrete} \\\\ \hline \text{DAPHNI} & \mathbf{-3.51} & -1.83 & \mathbf{-7.05} & \mathbf{-1.31} \\\\ \text{LCPO} & -4.39 & \mathbf{-1.49} & -8.92 & -1.76 \\\\ \text{SAC} & -15.45 & -10.95 & -16.86 & -9.02 \\\\ \text{OGD} & -4.78 & -3.43 & -9.14 & -4.83 \\\\ \text{MBCD} & -18.13 & -11.99 & -19.82 & -4.23 \\\\ \text{PT-DQN} & -14.73 & -13.21 & -15.80 & -7.15 \\\\ \end{array}$$ Notice that Online Cumulative Return remains our main metric, jointly reflecting forgetting resistance and online adaptation efficiency. DAPHNI achieves the highest mean return in all 20 MuJoCo settings. The MNRC results above directly assess historical retention, with DAPHNI ranking first in 10 of 16 wind settings and within the top two throughout. Together, these results demonstrate strong retention alongside efficient online adaptation. In the revised PDF, Section 3 defines MNRC and Appendix D.3 specifies the paired evaluation protocol. Section 5.2, Table 1 reports the comparison reproduced above, and Figures 3–4 provide the return and learning-curve results.

Weakness 2: Complementary mechanisms connect residual update alignment to retention of the complete policy.

DAPHNI combines three complementary mechanisms. The state-only base learns shared behavior from replayed experience; the residual modulator routes context-dependent updates according to contextual alignment; and KL controls the residual's deviation from the shared behavior. The inner product describes alignment in the chosen context representation, while the usefulness of that update structure is tested empirically. The experiments show localized residual influence on both Half Cheetah and Ant and measure historical return losses for the jointly updated policies. These experiments connect the residual update mechanism to the behavior of the complete architecture, including base updates and KL regulation.

In the revised PDF, Section 5.4 shows the residual influence patterns in Figures 6 and 11. Sections 5.2–5.3 and Appendix D.4 report historical return losses in Tables 1–2 and 7–9.

Question 1: Joint return and MNRC results explain the value of retaining the base policy.

The new comparison separates historical performance loss from overall learning progress. On Ant with sinusoidal wind, DAPHNI improves MNRC from SAC's −15.45-15.45\\% to −3.51-3.51\\%, while increasing return from 724.2724.2 to 1212.31212.3. The w/o base result illustrates why both measurements matter. Removing the shared branch confines updates to the similarity-driven modulator and eliminates a shared source of cross-context change, naturally explaining the smaller MNRC losses in some settings. The modulator must then also learn the shared behavior itself. On Half Cheetah under Random dynamics, w/o base improves MNRC from −5.82-5.82\\% to −2.20-2.20\\% while reducing return from 4604.14604.1 to 2202.32202.3. Read jointly, these results show that retaining the base supports much stronger behavior while maintaining strong retention, and DAPHNI also obtains the best ablation MNRC under Triangular and Discrete dynamics.

Mean ablation MNRC (%) on Half Cheetah. Rows and columns are transposed for readability; bold entries mark the best mean.

\text{Method} & \text{Sine} & \text{Triangular} & \text{Random} & \text{Discrete} \\\\ \hline \text{DAPHNI} & -2.78 & \mathbf{-1.13} & -5.82 & \mathbf{-0.31} \\\\ \text{w/o modulator} & -12.73 & -8.66 & -16.71 & -13.92 \\\\ \text{w/o KL} & -3.09 & -5.28 & -14.05 & -3.76 \\\\ \text{concat modulator} & -3.52 & -1.71 & -12.58 & -8.27 \\\\ \text{w/o base policy} & \mathbf{-1.60} & -1.24 & \mathbf{-2.20} & -0.84 \\\\ \text{inverted modulator} & -6.88 & -3.45 & -8.57 & -4.19 \\\\ \end{array}$$ In the revised PDF, Section 5.2 reports the Ant comparison. Section 5.3 interprets ablation returns and MNRC jointly; its Table 2 is reproduced above. **Question 2: Replay and KL coordinate an evolving shared base with context-specific adaptation.** The base remains adaptive because reusable behavior must also improve as experience accumulates. Its state-only input encourages a common state-to-action mapping, and replay trains this mapping using experience from encountered contexts. KL regulates how far the residual departs from this evolving shared behavior by constraining the current full policy relative to the current base policy. Historical experience enters both branches' training through replay. The new base-specific evidence is final-checkpoint extraction: the base outperforms the base from w/o KL in every setting on both Half Cheetah and Ant. Temporal retention is measured for the jointly updated full policy through MNRC. These complementary evaluations show useful shared behavior in the learned base and strong historical retention during joint adaptation. Mean evaluation returns on Half Cheetah. Rows and columns are transposed for readability; "base" denotes the extracted branch evaluated without further training. $$\begin{array}{lrrrrr} \text{Policy} & \text{Sine} & \text{Triangular} & \text{Random} & \text{Discrete} & \text{Multi} \\\\ \hline \text{DAPHNI base} & 2606.4 & 2491.2 & 2357.6 & 2830.4 & 1311.5 \\\\ \text{DAPHNI full} & 4044.7 & 4944.1 & 4604.1 & 5076.9 & 3452.8 \\\\ \text{state-only SAC} & 2247.9 & 2717.1 & 2327.0 & 3136.1 & 1523.4 \\\\ \text{w/o KL base} & 1856.5 & 1960.6 & 1579.1 & 2013.0 & 779.4 \\\\ \text{w/o KL full} & 3294.5 & 4323.3 & 3541.6 & 4056.6 & 2809.6 \\\\ \end{array}$$ In the revised PDF, Section 5.4, Table 3 reports the extraction comparison reproduced above, and Appendix D.4, Table 10 extends it to both benchmarks. Sections 3 and 5.2–5.3 define and evaluate temporal retention through MNRC.

Question 3: The analysis explains update propagation and experiments evaluate historical retention.

The residual mean analysis identifies a mechanism for limiting cross-context interference. For example, with orthogonal context representations, an update for one context has approximately zero first-order effect on the other context's residual mean under the stated approximation. This protects that component of the learned behavior while allowing adaptation elsewhere. The complete policy also includes the changing base and variance outputs, whose combined effect is evaluated through MNRC. Replacing the proposed interaction with concat or inverted modulation worsens MNRC in every tested wind setting on both benchmarks, with the influence probes supplying the corresponding update-pattern evidence. The theory and experiments therefore address successive links: how updates propagate, and how historical performance is retained.

In the revised PDF, Proposition 1 gives the residual mean analysis. Section 5.3, Table 2 (reproduced above) and Appendix D.4, Table 9 report the MNRC ablations; Figures 6 and 11 provide the update-pattern evidence.

Reviewer_yg1zOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the problem of catastrophic forgetting under continuously varying contexts in non-stationary reinforcement learning, and proposes DAPHNI that decomposes a Gaussian policy into a state-only base policy and a context-dependent residual modulator. They theoretically show that DAPHNI yields similarity-driven updates. Comparison experiments and ablation studies show consistent gains over representative baselines.

Critical strengths

This paper studies the problem of catastrophic forgetting under continuously varying contexts in non-stationary reinforcement learning, which is important and challenging for reinforcement learning agents bulit in real world. Although the "base + residual" decomposition itself is not new, the design of state-conditioned hypernetwork and KL divergence constraint is novel. The authors provided both theoritical and experimental analysis for their method.

Critical weaknesses

It seems that the scope of the theoretical results does not match the claimed contribution. Proposition 1 only characterizes the update direction of the residual mean. How does this support the conclusion of mitigating catastrophic forgetting.

Questions for authors

Can you explain (even heuristically) the respective theoretical roles of KL anchoring and the base policy in "forgetting mitigation"?

3 comments

Thank you for your thoughtful review and constructive questions about the connection between DAPHNI's design and forgetting mitigation.

Weakness: Direct retention evidence connects the residual update mechanism to forgetting mitigation.

The residual update analysis explains how an update propagates: under the NTK approximation, its effect on a probe context is scaled by the training–probe context inner product. Near-orthogonal representations suppress this source of interference. We have added direct historical-context evaluations to connect this mechanism to retention in the complete learning system.

Mean Negative Return Change (MNRC) is defined as follows: "Mean Negative Return Change (MNRC) directly measures the loss of previously acquired performance after adaptation to other contexts." For each of NN valid historical-context evaluation pairs, Rbefore,i>0R_{\mathrm{before},i}>0 and Rafter,iR_{\mathrm{after},i} are measured with matched initializations and the same context trajectory, before and after intervening training:

MNRC=1N∑i=1Nmin⁡(0,Rafter,i−Rbefore,iRbefore,i).\mathrm{MNRC}=\frac{1}{N}\sum_{i=1}^{N}\min\left(0,\frac{R_{\mathrm{after},i}-R_{\mathrm{before},i}}{R_{\mathrm{before},i}}\right).

Matching evaluation conditions isolates changes in historical performance. Clipping each pair before averaging makes gains contribute zero, so they cannot cancel losses; all valid pairs, including zeros, remain in the average. For example, changes of +20+20\\% and −10-10\\% give MNRC of −5-5\\%. MNRC is nonpositive, and values closer to zero indicate stronger forgetting resistance. The protocol uses deterministic evaluation from 10 fixed initial states, matched wind trajectories, within-seed averaging, and exclusion of the first 30K training steps from pair selection. The reported score averages valid seed-level scores across the 10 runs.

We evaluate MNRC only under the four wind dynamics because the paired protocol tests a historical high-wind region after training toward the opposite extreme; their ordered scalar context defines this comparison consistently. Multivariate contexts lack this unique ordering.

Mean MNRC (%; closer to zero is better), split by benchmark for readability. Bold entries mark the best mean.

Half Cheetah

\text{Method} & \text{Sine} & \text{Triangular} & \text{Random} & \text{Discrete} \\\\ \hline \text{DAPHNI} & \mathbf{-2.78} & -1.13 & \mathbf{-5.82} & \mathbf{-0.31} \\\\ \text{LCPO} & -3.01 & \mathbf{-1.02} & -9.19 & -0.34 \\\\ \text{SAC} & -12.73 & -8.66 & -16.71 & -13.92 \\\\ \text{OGD} & -3.14 & -3.60 & -8.39 & -2.51 \\\\ \text{MBCD} & -16.23 & -9.80 & -18.00 & -4.22 \\\\ \text{PT-DQN} & -12.80 & -11.27 & -15.57 & -6.87 \\\\ \end{array}$$ Hopper $$\begin{array}{lrrrr} \text{Method} & \text{Sine} & \text{Triangular} & \text{Random} & \text{Discrete} \\\\ \hline \text{DAPHNI} & -3.91 & -2.94 & \mathbf{-8.22} & -2.01 \\\\ \text{LCPO} & \mathbf{-3.48} & \mathbf{-2.81} & -10.26 & \mathbf{-1.65} \\\\ \text{SAC} & -10.86 & -8.60 & -12.60 & -9.16 \\\\ \text{OGD} & -5.90 & -5.53 & -10.35 & -6.33 \\\\ \text{MBCD} & -18.25 & -12.57 & -19.05 & -4.72 \\\\ \text{PT-DQN} & -14.96 & -13.19 & -18.88 & -8.48 \\\\ \end{array}$$ Walker2d $$\begin{array}{lrrrr} \text{Method} & \text{Sine} & \text{Triangular} & \text{Random} & \text{Discrete} \\\\ \hline \text{DAPHNI} & \mathbf{-4.14} & \mathbf{-2.19} & \mathbf{-7.54} & -2.27 \\\\ \text{LCPO} & -5.48 & -2.95 & -10.93 & \mathbf{-1.66} \\\\ \text{SAC} & -12.12 & -8.60 & -17.01 & -12.55 \\\\ \text{OGD} & -6.52 & -4.58 & -9.84 & -6.05 \\\\ \text{MBCD} & -15.75 & -10.30 & -19.07 & -5.64 \\\\ \text{PT-DQN} & -13.05 & -10.94 & -15.53 & -8.78 \\\\ \end{array}$$ Ant $$\begin{array}{lrrrr} \text{Method} & \text{Sine} & \text{Triangular} & \text{Random} & \text{Discrete} \\\\ \hline \text{DAPHNI} & \mathbf{-3.51} & -1.83 & \mathbf{-7.05} & \mathbf{-1.31} \\\\ \text{LCPO} & -4.39 & \mathbf{-1.49} & -8.92 & -1.76 \\\\ \text{SAC} & -15.45 & -10.95 & -16.86 & -9.02 \\\\ \text{OGD} & -4.78 & -3.43 & -9.14 & -4.83 \\\\ \text{MBCD} & -18.13 & -11.99 & -19.82 & -4.23 \\\\ \text{PT-DQN} & -14.73 & -13.21 & -15.80 & -7.15 \\\\ \end{array}$$ Notice that Online Cumulative Return remains our main metric, jointly reflecting forgetting resistance and online adaptation efficiency. DAPHNI achieves the highest mean return in all 20 MuJoCo settings. The MNRC results above directly assess historical retention, with DAPHNI ranking first in 10 of 16 wind settings and within the top two throughout. Together, these results demonstrate strong retention alongside efficient online adaptation.

The ablations below further connect the interaction design to retention: concat and inverted modulation worsen MNRC in every Half Cheetah wind setting. This pattern also holds on Ant, and the influence probes show concentrated update influence on matching and nearby contexts.

Mean ablation MNRC (%) on Half Cheetah. Rows and columns are transposed for readability; bold entries mark the best mean.

\text{Method} & \text{Sine} & \text{Triangular} & \text{Random} & \text{Discrete} \\\\ \hline \text{DAPHNI} & -2.78 & \mathbf{-1.13} & -5.82 & \mathbf{-0.31} \\\\ \text{w/o modulator} & -12.73 & -8.66 & -16.71 & -13.92 \\\\ \text{w/o KL} & -3.09 & -5.28 & -14.05 & -3.76 \\\\ \text{concat modulator} & -3.52 & -1.71 & -12.58 & -8.27 \\\\ \text{w/o base policy} & \mathbf{-1.60} & -1.24 & \mathbf{-2.20} & -0.84 \\\\ \text{inverted modulator} & -6.88 & -3.45 & -8.57 & -4.19 \\\\ \end{array}$$ Removing the base confines updates to the similarity-driven modulator and removes a shared source of cross-context change, naturally explaining the better MNRC of w/o base policy in some settings. The modulator must then also learn shared behavior: under Random dynamics, return falls from $4604.1$ to $2202.3$ despite MNRC improving from $-5.82\\%$ to $-2.20\\%$. Retaining the base supports much stronger behavior with strong retention. In the revised PDF, Proposition 1 gives the residual update analysis; Section 3 defines MNRC, and Appendix D.3 specifies its evaluation protocol. Sections 5.2–5.3 report the direct retention evaluations, with the comparison and ablation tables reproduced above corresponding to Tables 1 and 2. Section 5.2 provides the return results in Figures 3–4. Appendix D.4, Table 9 reports the Ant ablations, and Section 5.4 presents the influence probes in Figures 6 and 11.

Question: The base policy and KL anchoring support shared behavior and controlled residual adaptation.

The base policy provides reusable state-to-action behavior, while KL anchoring keeps context-specific residual corrections coordinated with that behavior. For example, locomotion can reuse a common movement pattern while adjusting actions to the current wind. KL penalizes residual mean deviations relative to the base variance and excessive variance modulation. The new extraction experiment shows that the base, without further training, attains 86–116% of independently trained state-only SAC's return and outperforms the base extracted from w/o KL by 27–68%. Removing KL also worsens MNRC across all eight Half Cheetah and Ant wind settings. These results support shared behavior in the base and controlled residual adaptation through KL as complementary parts of forgetting mitigation.

Mean evaluation returns on Half Cheetah. Rows and columns are transposed for readability; "base" denotes the extracted branch evaluated without further training.

\text{Policy} & \text{Sine} & \text{Triangular} & \text{Random} & \text{Discrete} & \text{Multi} \\\\ \hline \text{DAPHNI base} & 2606.4 & 2491.2 & 2357.6 & 2830.4 & 1311.5 \\\\ \text{DAPHNI full} & 4044.7 & 4944.1 & 4604.1 & 5076.9 & 3452.8 \\\\ \text{state-only SAC} & 2247.9 & 2717.1 & 2327.0 & 3136.1 & 1523.4 \\\\ \text{w/o KL base} & 1856.5 & 1960.6 & 1579.1 & 2013.0 & 779.4 \\\\ \text{w/o KL full} & 3294.5 & 4323.3 & 3541.6 & 4056.6 & 2809.6 \\\\ \end{array}$$ In the revised PDF, Section 4.4, Eq. (11) makes the KL role explicit. Section 5.4, Table 3 reports the extraction results reproduced above; Section 5.3 and Appendix D.4, Table 9 provide the MNRC ablations.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Common Response

We thank all reviewers for their careful assessments, constructive suggestions, and recognition of our work. We summarize the strengths highlighted in the reviews below.

Significance

  • Reviewer JwyL highlighted the important and underexplored problem of learning under continuously changing contexts without task boundaries.
  • Reviewers yg1z and rKYp emphasized its practical relevance, with yg1z also noting the challenge it poses for real-world reinforcement learning.

Novelty and simplicity

  • Reviewer yg1z identified the design of the state-conditioned hypernetwork and KL constraint as novel.
  • Reviewer JwyL described the architecture as simple and lightweight, noting its compatibility with SAC and context-inference methods.

Theoretical insights

  • Reviewer JwyL highlighted Proposition 1's interpretable account of how multiplicative state-context interactions shape residual updates and connect to cross-context interference.
  • Reviewer yg1z acknowledged the combination of theoretical and experimental analysis. JwyL also highlighted the component-wise ablations, base-policy extraction, and influence analysis.

Experimental results

  • Reviewer JwyL recognized the broad range of non-stationary settings, covering four MuJoCo environments and Lunar Lander, and later acknowledged that the additional experiments significantly improved the paper's empirical breadth.
  • Reviewer rKYp noted that the consistently higher average returns suggest potential practical benefits.

The constructive discussion also helped us refine the presentation:

  1. Evaluation metrics. We thank Reviewer JwyL for suggesting clearer evaluation reporting and clarify that Online Cumulative Return remains our main metric, reflecting overall performance through both retained behavior and online adaptation, while MNRC provides complementary historical-retention measurements.
  2. Additional experimental detail. Following Reviewer JwyL's suggestions, we supplemented baseline learning curves and included additional appendix results. We also evaluated the extracted base policy's retention during continued joint training, as suggested by Reviewer rKYp.
  3. Clearer notation and evaluation descriptions. Following Reviewer JwyL's comments, we clarified the return calculation, distinguished environment steps from theoretical training time, and made the evaluation protocol more explicit.

Please refer to the latest revised PDF, which addresses questions about the figures in earlier versions.

Thank you again for your time, thoughtful feedback, and constructive engagement throughout the rebuttal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.