- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes DAPHNI, a policy architecture for mitigating catastrophic forgetting in RL with continuously varying contexts. The Gaussian policy is decomposed into a state-only base policy and a context-dependent residual produced by a state-conditioned hypernetwork, with a KL penalty anchoring the final policy to the base. Under an NTK approximation, the authors show that the residual update at one context is proportional to its inner product with the training context. Experiments on MuJoCo benchmarks and Lunar Lander report higher returns than several baselines.
Critical strengths
Strengths
-
The paper addresses an important and underexplored problem: catastrophic forgetting in RL when the context changes continuously, without task boundaries.
-
The proposed architecture is simple and lightweight. It adds only a base policy, a hypernetwork-based residual, and a KL term, so it can be easily attached to off-policy algorithms such as SAC and combined with context-inference methods (Appendix C).
-
Proposition 1 offers an interpretable account of how the multiplicative state-context interaction shapes residual updates, linking the architectural choice to cross-context interference.
-
The evaluation covers a broad range of non-stationary settings, including four MuJoCo environments with five context dynamics (smooth, noisy, abrupt, and multivariate) and Lunar Lander with simultaneous transition and reward changes.
-
The ablation study examines each component (modulator, KL constraint, base policy, and interaction type), and the authors make an effort to probe the underlying mechanism through base policy extraction and an influence analysis.
-
The KL coefficient is tuned automatically via a target-KL dual update, and the sensitivity analysis (Appendix B.2) shows that performance is fairly robust to the choice of target KL.
Critical weaknesses
-
Insufficient evidence of forgetting mitigation. The only forgetting-specific metric, CRC (Tables 1 and 7), is reported only for DAPHNI and its ablations, not for any baseline, and only on Half Cheetah. Final returns (Figure 4, Table 4) cannot distinguish less forgetting from faster learning, and the smoothed, seed-averaged curves in Figure 3 (with independently sampled wind phases per seed) can hide drops when a previous context recurs. CRC itself is also questionable: it is positive for every method (+5.5% to +27.1%), so it conflates retention with continued learning, and its normalization by R_before favors methods with low initial performance. More direct evidence is needed, such as CRC for the baselines or periodic evaluation at fixed contexts during training.
-
Fairness and coverage of the baselines. PT-DQN is applied via action discretization and MBCD without its model-based rollouts (Appendix D.2), so their poor performance may reflect these modifications rather than forgetting. The comparison also omits the alternatives closest to DAPHNI's components, such as a hypernetwork-based context-conditioned SAC or SAC regularized toward an information-restricted default policy.
-
Ablations limited to Half Cheetah. All ablations and analyses (Tables 6-8, Figure 6) are conducted only on Half Cheetah. Since the paper claims larger gains in higher-dimensional environments such as Ant, ablations there are needed to show that the findings generalize.
-
Weak analysis of the base-residual decomposition. The extracted base policy is compared only with a random policy (Tables 2 and 8), which is an uninformative reference; a state-only SAC policy would better test whether the base captures shared behavior. Moreover, the extracted base reaches only about 38-64% of the full DAPHNI return, implying a large residual, which conflicts with the premise in Section 4.1 that the context-specific component is small and localized.
Minor
-
Evaluation metric. J_eval in Eq. 1 is a discounted return, but with gamma = 0.99 the reported values in the thousands (e.g., 4044.7 and 5076.9 on Half Cheetah) appear to be undiscounted sums. In addition, Section 3 states that online cumulative return is the primary metric, whereas the results are measured in separate evaluation episodes with a deterministic policy (Table 11, Figure 3, Appendix D.3). Please clarify how the reported metric is computed.
-
Notation. The symbol t denotes the environment time step in Section 3, Eq. 1, and Algorithm 1, but denotes training (gradient-flow) time in Proposition 1 and Appendix A without being redefined. Similarly, T is the episode horizon in Eq. 1 but the training-time interval in Lemma 1. Using a separate symbol for training time would avoid confusion.
Questions for authors
Questions
-
Relation to prior KL-regularized methods. Anchoring the policy to a state-only base policy via a KL penalty resembles prior approaches such as learned default policies with restricted inputs [1] and Distral [2]. How does DAPHNI differ from these methods?
-
Missing baseline curves in Figure 3. Could the authors provide learning curves for all baselines, at least in the appendix? In particular, OGD, identified in Figure 7 as the strongest baseline under the discrete wind form (4413.4 in Table 4), is missing from the discrete panel of Figure 3.
-
Context recurrence and training length. What are the periods of the wind schedules, and how often does each context region recur within 100K steps? Since several curves in Figure 3 are still rising, longer training would help clarify whether the gains reflect mitigated forgetting or faster early learning.
-
Relation to hypernetwork-based context conditioning. Prior work [3] has explored both directions of combining state and context with hypernetworks: Hyper-MAML uses the context as the hypernetwork input, while its PEARL critic uses the state as the hypernetwork input with the context fed to the generated network. Could the authors discuss how the design in Section 4.2 relates to these prior choices?
[1] Information Asymmetry in KL-Regularized RL, ICLR 2019
[2] Distral: Robust Multitask Reinforcement Learning, NeurIPS 2017
[3] Recomposing the Reinforcement Learning Building Blocks with Hypernetworks, ICML 2021
Suggestions
-
Clarifying Proposition 1. Please define z_A and z_B at their first appearance (page 2) and state that the proportionality to z_A^T z_B holds with s, s', and z_B fixed. It would also help to clarify what the "current context z_B" means when each replay batch mixes many contexts, and whether the inner product remains a meaningful similarity for the raw context coordinates used in the multivariate and Lunar Lander settings.
-
Positioning with respect to meta-RL and contextual RL. The related work treats meta-RL mainly as a context-inference tool, although meta-RL and contextual RL are naturally formulated as CMDPs. Discussing related lines of work, such as continuous adaptation in nonstationary environments [4], zero-shot generalization across contexts [5], and hypernetwork policies conditioned on known context such as HyperZero [6], would clarify the contribution.
-
Relation to shared-backbone adaptation. The motivation in Section 4.1, a shared policy component with lightweight context-specific adaptation, resonates with recent work such as APB [7]. The authors may find it useful to discuss how DAPHNI's residual relates to such shared-backbone formulations.
[4] Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments, ICLR 2018
[5] A Survey of Zero-Shot Generalisation in Deep Reinforcement Learning, JAIR 2023
[6] Hypernetworks for Zero-Shot Transfer in Reinforcement Learning, AAAI 2023
[7] Adaptive Policy Backbone via Shared Network, ICML 2026
12 comments
Weakness 2: Baseline adaptations and component comparisons clarify the scope of the evidence.
PT-DQN retains permanent–transient value decomposition with branching action discretization, while MBCD retains probabilistic context detection and policy reuse and trains policies from real transitions. We make these continuous-action benchmark adaptations explicit, and assess retention additionally against continuous-action SAC, LCPO, and OGD. DAPHNI combines higher returns with strong MNRC against these baselines. For the closest components, we compare the hypernetwork modulator without a base, concat and inverted modulators, and w/o KL, and add independently trained state-only SAC and base extraction from w/o KL. These comparisons examine decomposition, interaction, and anchoring directly. The relationship to learned-default-policy methods is addressed in our response to Question 1.
Mean ablation MNRC (%) on Half Cheetah. Rows and columns are transposed for readability; bold entries mark the best mean.
\text{Method} & \text{Sine} & \text{Triangular} & \text{Random} & \text{Discrete} \\\\ \hline \text{DAPHNI} & -2.78 & \mathbf{-1.13} & -5.82 & \mathbf{-0.31} \\\\ \text{w/o modulator} & -12.73 & -8.66 & -16.71 & -13.92 \\\\ \text{w/o KL} & -3.09 & -5.28 & -14.05 & -3.76 \\\\ \text{concat modulator} & -3.52 & -1.71 & -12.58 & -8.27 \\\\ \text{w/o base policy} & \mathbf{-1.60} & -1.24 & \mathbf{-2.20} & -0.84 \\\\ \text{inverted modulator} & -6.88 & -3.45 & -8.57 & -4.19 \\\\ \end{array}$$ Removing the base confines updates to the similarity-driven modulator and removes a shared source of cross-context change, naturally explaining the better MNRC of w/o base policy in some settings. The modulator must then also learn shared behavior: under Random dynamics, return falls from $4604.1$ to $2202.3$ despite MNRC improving from $-5.82\\%$ to $-2.20\\%$. Retaining the base supports much stronger behavior with strong retention. In the revised PDF, Appendix D.2 specifies the baseline adaptations; Section 5.2 reports the comparison evidence in Figures 3–4 and Table 1. Section 5.3 gives the component ablations, including Table 2 reproduced above, and Section 5.4 provides the new extraction comparisons.Weakness 3: Ant ablations and mechanism studies extend the component findings to quadruped control.
We extended both ablation and mechanism studies to Ant. DAPHNI achieves the highest mean return under all five Ant dynamic forms and the best ablation MNRC under Triangular, Random, and Discrete dynamics. Its extracted base outperforms the base from w/o KL in every Ant setting, and the interaction probe reproduces the concentration of update influence on matching and nearby contexts. The component findings therefore extend to higher-dimensional quadruped control.
Mean ablation returns on Ant. Bold entries mark the best mean.
\text{Method} & \text{Sine} & \text{Triangular} & \text{Random} & \text{Discrete} & \text{Multi} \\\\ \hline \text{DAPHNI} & \mathbf{1212.3} & \mathbf{958.5} & \mathbf{817.7} & \mathbf{1087.8} & \mathbf{1107.7} \\\\ \text{w/o modulator} & 724.2 & 710.9 & 523.2 & 836.4 & 708.2 \\\\ \text{w/o KL} & 1013.1 & 807.8 & 594.0 & 765.5 & 844.2 \\\\ \text{concat modulator} & 932.9 & 755.1 & 500.5 & 930.7 & 802.4 \\\\ \text{w/o base policy} & 850.2 & 754.1 & 399.0 & 737.6 & 621.0 \\\\ \text{inverted modulator} & 1045.3 & 823.8 & 672.5 & 905.0 & 922.9 \\\\ \end{array}$$ Mean ablation MNRC (%) on Ant; closer to zero is better. Bold entries mark the best mean. $$\begin{array}{lrrrr} \text{Method} & \text{Sine} & \text{Triangular} & \text{Random} & \text{Discrete} \\\\ \hline \text{DAPHNI} & -3.51 & \mathbf{-1.83} & \mathbf{-7.05} & \mathbf{-1.31} \\\\ \text{w/o modulator} & -15.45 & -10.95 & -16.86 & -9.02 \\\\ \text{w/o KL} & -4.95 & -7.38 & -16.72 & -5.14 \\\\ \text{concat modulator} & -6.39 & -4.72 & -17.33 & -6.79 \\\\ \text{w/o base policy} & \mathbf{-1.76} & -1.99 & -7.47 & -1.73 \\\\ \text{inverted modulator} & -8.47 & -3.48 & -9.70 & -4.32 \\\\ \end{array}$$ As in Half Cheetah, removing the shared branch confines updates to the similarity-driven modulator, naturally explaining the better Sine MNRC of w/o base policy. Its Sine return falls from $1212.3$ to $850.2$, while DAPHNI leads both return and MNRC under the other three wind dynamics. Mean base policy extraction returns on Ant; "base" denotes the extracted branch evaluated without further training. $$\begin{array}{lrrrrr} \text{Policy} & \text{Sine} & \text{Triangular} & \text{Random} & \text{Discrete} & \text{Multi} \\\\ \hline \text{DAPHNI base} & 756.5 & 563.5 & 412.2 & 624.0 & 454.6 \\\\ \text{DAPHNI full} & 1212.3 & 958.5 & 817.7 & 1087.8 & 1107.7 \\\\ \text{state-only SAC} & 670.5 & 518.4 & 457.4 & 723.2 & 492.5 \\\\ \text{w/o KL base} & 564.3 & 355.0 & 287.6 & 390.8 & 227.2 \\\\ \text{w/o KL full} & 1013.1 & 807.8 & 594.0 & 765.5 & 844.2 \\\\ \end{array}$$ In the revised PDF, Sections 5.3–5.4 point to the Ant results in Appendix D.4. The three tables reproduced above give the Ant results from Table 8 (ablation returns), Table 9 (ablation MNRC), and Table 10 (base extraction); Figure 11 shows the hypernetwork interaction probe. **Weakness 4: Extraction controls support shared behavior and the contribution of KL anchoring.** The new extraction comparison directly uses independently trained state-only SAC. DAPHNI's extracted base achieves 86–116% of its return without additional training, and exceeds the base extracted from w/o KL by 27–68%. The comparison also extends to Ant. This supports useful shared behavior and the role of KL in maintaining that behavior. The base/full return ratio measures the outcome of executing two policies, not the magnitude of their action differences: even one changed decision near a trap can determine whether a goal is reached. The local decomposition motivates the architecture without requiring globally small corrections. Full DAPHNI's additional return demonstrates the value of context-specific adaptation on top of the shared behavior. Mean evaluation returns on Half Cheetah. Rows and columns are transposed for readability; "base" denotes the extracted branch evaluated without further training. $$\begin{array}{lrrrrr} \text{Policy} & \text{Sine} & \text{Triangular} & \text{Random} & \text{Discrete} & \text{Multi} \\\\ \hline \text{DAPHNI base} & 2606.4 & 2491.2 & 2357.6 & 2830.4 & 1311.5 \\\\ \text{DAPHNI full} & 4044.7 & 4944.1 & 4604.1 & 5076.9 & 3452.8 \\\\ \text{state-only SAC} & 2247.9 & 2717.1 & 2327.0 & 3136.1 & 1523.4 \\\\ \text{w/o KL base} & 1856.5 & 1960.6 & 1579.1 & 2013.0 & 779.4 \\\\ \text{w/o KL full} & 3294.5 & 4323.3 & 3541.6 & 4056.6 & 2809.6 \\\\ \end{array}$$ In the revised PDF, Section 4.2 gives the local decomposition. Section 5.4, Table 3 reports the extraction results reproduced above, and Appendix D.4, Table 10 extends the comparison to Ant.
Thank you for your detailed review and concrete suggestions for improving the retention evidence, component analysis, and positioning of DAPHNI.
Weakness 1: Matched historical-context evaluations separate return losses from continued improvement.
We have replaced CRC with MNRC and expanded historical-context evaluation to every comparison baseline on all four MuJoCo benchmarks. The definition and results are reproduced below.
Mean Negative Return Change (MNRC) directly measures historical performance loss after adaptation elsewhere. For N valid pairs, returns Rbefore,i>0 and Rafter,i are measured before and after intervening training with matched initializations and the same context trajectory:
MNRC=N1∑i=1Nmin(0,Rbefore,iRafter,i−Rbefore,i).
Matched evaluations isolate historical performance changes. Gains are clipped to zero before averaging, so they cannot offset losses; all valid pairs, including zeros, count. Changes of +20 and −10 give −5 MNRC. MNRC is nonpositive; closer to zero means stronger retention. We use deterministic evaluation from 10 fixed initial states and matched wind trajectories, excluding the first 30K steps from pair selection. We average pairs within each seed, then valid seed scores across 10 runs.
We evaluate MNRC only under the four wind dynamics because the paired protocol tests a historical high-wind region after training toward the opposite extreme; their ordered scalar context defines this comparison consistently. Multivariate contexts lack this unique ordering.
For a fixed absolute loss, a smaller positive reference return makes MNRC more negative; positive gains receive zero irrespective of the denominator. The 30K-step exclusion avoids unstable early reference policies. Scores are computed from paired evaluations within each seed, independently of learning-curve smoothing or cross-seed wind phases, and are read alongside absolute returns.
Mean MNRC (%); closer to zero is better. Bold marks the best mean.
Half Cheetah
\text{Method} & \text{Sine} & \text{Triangular} & \text{Random} & \text{Discrete} \\\\ \hline \text{DAPHNI} & \mathbf{-2.78} & -1.13 & \mathbf{-5.82} & \mathbf{-0.31} \\\\ \text{LCPO} & -3.01 & \mathbf{-1.02} & -9.19 & -0.34 \\\\ \text{SAC} & -12.73 & -8.66 & -16.71 & -13.92 \\\\ \text{OGD} & -3.14 & -3.60 & -8.39 & -2.51 \\\\ \text{MBCD} & -16.23 & -9.80 & -18.00 & -4.22 \\\\ \text{PT-DQN} & -12.80 & -11.27 & -15.57 & -6.87 \\\\ \end{array}$$ Hopper $$\begin{array}{lrrrr} \text{Method} & \text{Sine} & \text{Triangular} & \text{Random} & \text{Discrete} \\\\ \hline \text{DAPHNI} & -3.91 & -2.94 & \mathbf{-8.22} & -2.01 \\\\ \text{LCPO} & \mathbf{-3.48} & \mathbf{-2.81} & -10.26 & \mathbf{-1.65} \\\\ \text{SAC} & -10.86 & -8.60 & -12.60 & -9.16 \\\\ \text{OGD} & -5.90 & -5.53 & -10.35 & -6.33 \\\\ \text{MBCD} & -18.25 & -12.57 & -19.05 & -4.72 \\\\ \text{PT-DQN} & -14.96 & -13.19 & -18.88 & -8.48 \\\\ \end{array}$$ Walker2d $$\begin{array}{lrrrr} \text{Method} & \text{Sine} & \text{Triangular} & \text{Random} & \text{Discrete} \\\\ \hline \text{DAPHNI} & \mathbf{-4.14} & \mathbf{-2.19} & \mathbf{-7.54} & -2.27 \\\\ \text{LCPO} & -5.48 & -2.95 & -10.93 & \mathbf{-1.66} \\\\ \text{SAC} & -12.12 & -8.60 & -17.01 & -12.55 \\\\ \text{OGD} & -6.52 & -4.58 & -9.84 & -6.05 \\\\ \text{MBCD} & -15.75 & -10.30 & -19.07 & -5.64 \\\\ \text{PT-DQN} & -13.05 & -10.94 & -15.53 & -8.78 \\\\ \end{array}$$ Ant $$\begin{array}{lrrrr} \text{Method} & \text{Sine} & \text{Triangular} & \text{Random} & \text{Discrete} \\\\ \hline \text{DAPHNI} & \mathbf{-3.51} & -1.83 & \mathbf{-7.05} & \mathbf{-1.31} \\\\ \text{LCPO} & -4.39 & \mathbf{-1.49} & -8.92 & -1.76 \\\\ \text{SAC} & -15.45 & -10.95 & -16.86 & -9.02 \\\\ \text{OGD} & -4.78 & -3.43 & -9.14 & -4.83 \\\\ \text{MBCD} & -18.13 & -11.99 & -19.82 & -4.23 \\\\ \text{PT-DQN} & -14.73 & -13.21 & -15.80 & -7.15 \\\\ \end{array}$$ Notice that Online Cumulative Return remains our main metric, jointly reflecting forgetting resistance and online adaptation efficiency. DAPHNI achieves the highest mean return in all 20 MuJoCo settings. The MNRC results above directly assess historical retention, with DAPHNI ranking first in 10 of 16 wind settings and within the top two throughout. Together, these results demonstrate strong retention alongside efficient online adaptation. In the revised PDF, Section 3 defines MNRC and Appendix D.3 specifies the evaluation protocol. Section 5.2, Table 1 reports the results reproduced above, and Figures 3–4 provide the return results. Appendix D.4, Table 7 supplies the corresponding seed-level standard deviations.