acceptodds
Under review as a conference paper at ICLR 2027

Space-sampled Value Decay: Forgetting Mechanisms for Non-stationary Reinforcement Learning

Abstract

Reinforcement Learning agents deployed on physical systems must adapt continually, since degradation and shifting environment conditions change the dynamics (they drift) over time. In the hardest version of this problem, the agent interacts with a single system that might drift at every timestep, leaving no opportunity to revisit past conditions – a setting we call Single Environment, One-Shot Non-Stationary Reinforcement Learning (SEOS-NSRL). We argue that this setting calls for selective forgetting rather than re-learning, and introduce Space-sampled Value Decay (SsVD), which pulls value estimates of randomly chosen elements of the state space to a baseline value, so that outdated information in non visited regions is discarded. SsVD does not require resetting or change-point detection and plugs into modern off-policy algorithms; we integrate it into Soft Actor Critic and Deep Q-Networks. Across 6 non-stationary environments, SsVD improves upon its direct base algorithms and attains the best mean rank across all. The SsVD mechanism can also induce optimism which we show on hard-exploration tasks, although we investigate the connection here only briefly.

Primary area:
reinforcement learning
Keywords:
Non-stationary Reinforcement Learning, Continual Reinforcement Learning, Drift, Forgetting, Exploration

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_AfsLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Space-sampled Value Decay (SsVD), which regularizes value estimates at sampled states toward a baseline to support non-stationary adaptation. It integrates the mechanism into DQN and SAC and provides a tabular theoretical analysis. Experiments on six non-stationary environments and three exploration tasks report improvements over the corresponding base algorithms.

Critical strengths
  1. The paper combines theoretical insights with empirical evaluation. The experimental results support the intuition behind value decay and provide evidence of its practical usefulness.
  2. The method adds a single auxiliary loss to the standard training objective. This makes it straightforward to implement.
Critical weaknesses
  1. The method uses environment-specific values of η\eta and bb, but their selection procedure is insufficiently explained. Please clarify how these parameters should be chosen before deployment and provide sensitivity analysis.

  2. The experiments do not establish a clear overall advantage over PPO, and the evidence for substantial gains on SAC is limited. Further analysis would help clarify when SsVD provides a practically meaningful improvement over these baselines.

  3. The non-stationary evaluation focuses on relatively simple tasks and a limited set of drift patterns. Testing more challenging environments and more diverse reward and dynamics shifts would strengthen the generality of the conclusions.

I would be happy to raise my score if the authors can adequately address these three concerns.

Questions for authors
  1. How should η\eta and bb be selected in practice—for example, for humanoid control in a continually changing environment with unknown future drift? In such a setting, what would motivate choosing SsVD over PPO directly?

  2. How do the results change when the UTD ratio of DQN/SAC is increased? Do η\eta and bb require recalibration, and does SsVD retain its benefits when the base agents receive comparable UTD tuning?

  3. Could a similar mechanism improve PPO, whose value estimates may also become outdated despite its on-policy updates? In particular, could an optimistic value prior improve performance, and how would it interact with advantage estimation and policy learning?

5 comments

Dear Reviewer AfsL, we highly appreciate your input and openness to participate in the rebuttal process! We thank you for highlighting how our experiments confirm the intuition behind our proposed mechanism.

Scaling to higher dim. control

The non-stationary evaluation focuses on relatively simple tasks and a limited set of drift patterns. Testing more challenging environments and more diverse reward and dynamics shifts would strengthen the generality of the conclusions.

We extend our existing set of different drift patterns (ChooseBox and MultiPadLunarLander have changing rewards) by utilizing two different sine based periodic drifting characteristic for the higher-dimensional Walker2D environment. For this we adapt the original sampling scheme of SsVD (Appendix B.4).

This is now a new section 4.4 in our revised document. Table 2 clearly shows that PPO fails and LILAC falls behind considerably in the considered problem setting. SsVD is competitive for slow drift but learns more slowly in case of fast periodic drift. Following your suggestion we also included PPO-F (see below).

We improved the already existing discussion of this SsVD's limitation: we leave further modifications for higher dimensional state-spaces and image based observations as future work.

Could a similar mechanism improve PPO, whose value estimates may also become outdated despite its on-policy updates? In particular, could an optimistic value prior improve performance, and how would it interact with advantage estimation and policy learning?

To test whether the mechanism carries over to on-policy methods, we implemented PPO-F. It adds the SsVD term to PPO's state-value critic.

EnvironmentPPOPPO-F
InvertedPendulum (10 seeds)863 ± 89859 ± 111
Walker2d, slow mass drift (5 seeds)925 ± 701426 ± 182
Walker2d, fast mass drift (5 seeds)1251 ± 1221417 ± 188

On Walker2d, PPO-F improves PPO's mean online return by factors 1.1x/1.5x depending on the chosen drift speed, so SsVD can be helpful for PPO. On InvertedPendulum, however it does not fix the case where PPO fails to track the optimal policy late in the experiment. We added PPO-F to Table 2 and leave further analysis for on-policy algorithms to future work.

Sensitivity to hyper-parameters

The method uses environment-specific values of η\eta and bb, but their selection procedure is insufficiently explained. Please clarify how these parameters should be chosen before deployment and provide sensitivity analysis.

For the baseline bb see our answer to Reviewer pvC9 or Figure 4 b) (tldr: choose the midpoint). Figure C.3 a) + b) shows a sensitivity analysis with respect to η\eta and ξ\xi. We find SsVD to be robust to these choices in most tested cases except for MountainCar (too high values for η\eta degrade performance) and on Walker2D, thus higher-dimensional tasks may require tuning η\eta and ξ\xi.

How do the results change when the UTD ratio of DQN/SAC is increased? Do and require recalibration, and does SsVD retain its benefits when the base agents receive comparable UTD tuning?

This ablation is adressed in Figure 4(c): for the selected environments we find that UTD to not have a significant effect - it neither improves the base algorithms nor does it hurt SsVD. Only for the newly introduced Walker2D we observe SAC improvement for a higher UTD that we also, but only for even higher UTD, observe for SsVD - we ultimately believe this to be environment dependent.

Recalibrating η\eta with the UTD ratio (dividing it by the UTD factor) did not help on any environment, so we keep the parameters fixed.

Thank you for your response and the additional experiments, including Walker2D, PPO-F, and the sensitivity analyses.


I. Walker2D results: The improvement over SAC under slow drift is only about 1.7%, which is small relative to the substantial variability across seeds. Under fast drift, SAC-F performs worse on average. These results therefore do not establish a clear practical benefit over SAC in the newly added control task.


II. Hyperparameter selection under unknown drift: The midpoint heuristic for b and the sensitivity analyses are helpful. However, it remains unclear how to select b, η, and ξ before deployment when future drift is unknown. In particular, the paper does not clearly establish whether parameters selected independently of the test drift conditions retain their benefits across unseen drift speeds or patterns.


I appreciate the authors’ efforts, but these concerns remain insufficiently addressed. I am therefore inclined to maintain my rating of weak rejection.

Thank you for the careful reading and for engaging constructively throughout the discussion phase! We would like to clarify precisely what the new experiments establish regarding the scope of our claims and pre-deployment hyperparameter selection:

I. Role of Walker2D & Practical Benefit Over Base Algorithms We agree that SAC-F matches rather than outperforms vanilla SAC on the periodic Walker2D task, and we intentionally present Section 4.4 ("Limits of scaling to higher-dimensional control") to transparently document the boundary of naive uniform sampling in unbounded 17D state spaces (Appendix B.4). Two points contextualize this result within the paper's broader findings:

  • Core empirical claim on bounded spaces: Where valid state-space sampling is well-defined - our six primary non-stationary benchmarks -SsVD consistently improves over its direct base algorithm (DQN and SAC) in 6 out of 6 cases (Figure 3, right), including substantial gains for SAC-F when drift permanently invalidates buffered experience.
  • The mechanism still transfers to 17D control: Even with our simple bounding-box sampling heuristic on Walker2D, applying SsVD with the exact same (b,η,ξ)(b, \eta, \xi) parameters (no retuning) improves PPO under both drift speeds and on every single seed: 1426 ± 182 vs. 925 ± 70 (slow) and 1417 ± 188 vs. 1251 ± 122 (fast). Furthermore, restricting decay samples purely to the replay buffer degrades SAC-F on Walker2D just as it does on five of the six primary benchmarks (Figure 4a), confirming that out-of-buffer value decay is a consistent principle worth extending with manifold-aware samplers in future work.

II. Selecting bb, η\eta, and ξ\xi Before Deployment Rather than tuning hyperparameters to test drift profiles, our evaluation demonstrates that fixed, pre-deployment defaults retain their benefits across unseen drift directions, speeds, and even the absence of drift:

  • bb depends only on the known reward range and γ\gamma, not the drift: The neutral midpoint heuristic (b≈Vmin⁡+Vmax⁡2b \approx \frac{V_{\min}+V_{\max}}{2}, Figure 4b) requires only design-time reward bounds and γ\gamma - and in practice, even a simple b=0b=0 default often sufficed without tuning (Table C.2).
  • Wide defaults and insensitivity for η\eta and ξ\xi: Across all 7 benchmarks in Table C.2, we use only two coarse settings: ξ∈0.05,0.25\xi \in {0.05, 0.25} and η∈0.0015,0.015\eta \in {0.0015, 0.015}. On lower-dimensional tasks where the state space is directly sampled, performance is flat across ξ∈[0.01,0.5]\xi \in [0.01, 0.5] and across an order-of-magnitude variation in η\eta (Figure C.3a–b).
  • No-retuning transfer across unseen drift regimes (including zero and reversed drift):
    1. Stochastic trajectories: Probabilistic schedulers generate distinct drift trajectories per seed under fixed (b,η,ξ)(b, \eta, \xi) (Figure B.2).
    2. Unseen "no-drift" deployment: Keeping (b,η,ξ)(b, \eta, \xi) completely unchanged when drift is removed entirely (stationary ablation, Figure B.5) matches or improves over vanilla DQN/SAC; clearly showing no penalty if anticipated drift does not occur.
    3. New experiment: Unseen reversed (increasing) drift: To test transfer to an unseen drift pattern directly, we evaluated CartPole and InvertedPendulum under increasing parameter drift (reversing the decreasing drift in the main paper) while keeping bb, η\eta, and ξ\xi strictly identical to Table C.2 (mean online return ± sample std over 10 seeds):
EnvironmentDriftLimitedBaseSsVD
CartPole (DQN)reversed (force 10 → 50)72 ± 9983 ± 10288 ± 25
CartPole (DQN)paper (force 10 → 0.9)49 ± 1599 ± 68394 ± 12
InvertedPendulum (SAC)reversed (cart mass 10.5 → 31.4)950 ± 101910 ± 14984 ± 4
InvertedPendulum (SAC)paper (cart mass 10.5 → 0.5)390 ± 92945 ± 13985 ± 3

Under both held-out increasing drifts - with all SsVD settings unchanged - SsVD remains clearly best. Note that on InvertedPendulum, increasing cart mass is milder on the initial policy than decreasing it (a policy frozen at 100k steps still averages 950 ± 101), yet continued off-policy training under shifting mass degrades vanilla SAC to 910 ± 14 whereas SsVD stabilizes learning at near-optimal return (984 ± 4) with minimal variance.

Together, these results show that for bounded observation spaces, practitioners can select (b,η,ξ)(b, \eta, \xi) a priori from reward bounds and standard coarse defaults, retaining strong performance across decreasing, increasing, and stationary regimes. In high-dimensional unbounded spaces (Walker2D), sensitivity increases alongside the state-sampling approximation, which we highlight transparently in Section 4.4 and Section 6 as an avenue for future adaptive sampling methods.

Reviewer_aq7COpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Space-sampled Value Decay (SsVD), a forgetting mechanism for reinforcement learning in environments whose dynamics or rewards change over time. SsVD samples states independently of the agent’s collected experience and uses an auxiliary loss to pull their action-value estimates toward a chosen baseline. The intended effect is to reduce reliance on outdated estimates, including in regions the agent no longer visits. The method is integrated into DQN and SAC and evaluated across six non-stationary settings, with comparisons against the base algorithms, PPO, periodic resets, and L2-to-initialization regularization. The paper also provides a theoretical analysis of the isolated decay update and explores how optimistic baseline values can encourage exploration in stationary tasks.

Critical strengths
  • The idea is simple and relatively easy to integrate into existing algorithms, without requiring task identifiers or explicit change detection.

  • SsVD improves mean online returns over the corresponding base agents in all non-stationary settings, with substantial gains in some cases. The stationary controls and value-function visualizations help interpret these results.

Critical weaknesses
  • The experiments do not isolate the benefit of sampling from the full state space. Comparing the same loss on replay-buffer states would help establish whether the proposed sampling strategy matters beyond adding regularization.

  • The comparisons leave the method’s competitiveness unclear. Reactive Exploration, SF+SC, and LILAC study related forms of non-stationarity and would be relevant alternatives. PPO already performs strongly in the reported experiments, while SsVD’s advantage in average rank is not statistically significant.

Questions for authors
  • How does performance change when the auxiliary states come from the replay buffer or recent experience?

  • How sensitive are the results to the baseline bb, particularly when it encourages exploration?

  • Could the authors include comparisons with the methods above and measure recovery after changes, alongside average returns over training?

4 comments

Dear Reviewer aq7C, we thank you for the time spent with our submission! Thank you for highlighting our visualization efforts to make the results interpretable.

How does performance change when the auxiliary states come from the replay buffer or recent experience?

We add this experiment in Figure 4 a): we clearly find that sampling from the replay buffer to be significantly worse in 5/7 cases, in the others sampling from the state-space is still favorable, but only slightly. So sampling from the state space is a key element in our method.

How sensitive are the results to the baseline , particularly when it encourages exploration?

For the non-stationary case see the answer to Reviewers pvC9/AfsL.

For exploration: we added this experiment as Figure C.3 in the Appendix. On simpler environments such as RiverSwim a higher baseline makes it insensitive to the other parameters. On MountainCarContinuous we find that the baseline needs to be optimistic, but also hurting the performance if chosen too optimistic.

Could the authors include comparisons with the methods above and measure recovery after changes, alongside average returns over training?

We include comparisons with "LILAC" and "ReactiveExploration" (and a distribution robust RL variant "R-cont." by suggestion of Reviewer pvC9). Full results are in Table C.1 and discussed in the "Further Baselines" paragraph in section 4.1.

We find LILAC to not transfer to the settings tested here, performing worse than base SAC (we sanity checked LILACs implementation on the HalfCheetah experiment of the original paper). "ReactiveExploration" provides similiar/better performance than PPO which its based on; yet it doesnt help on ChooseBox or InvertedPendulum where we observe both to fail. To be clear, as stated in Section 4.1, the rank differences are not statistically significant; the consistent gain over the base learner in all six environments, Figure 3 right, is our main claim.

Recovery: under Assumption 1 the agent is not informed about change points and most of our drifts are gradual, so recovery times are not well defined. The online return over training captures adaptation instead.

Dear Reviewer aq7C,

Thank you again for your review. In our rebuttal and revised paper we added comparisons to the suggested additional baselines, illustrating the competitiveness of our approach. As suggested, we found that sampling purely from the replay buffer significantly degrades performance, confirming that full state space sampling is key to our method.

We would be grateful to know whether our responses resolve your concerns. If so, we would appreciate it if you would consider updating your assessment; if not, we would welcome the chance to address any remaining issues before the approaching deadline.

Best regards and many thanks for your time, The Authors

Thank you for the additional experiments. The replay-buffer ablation and new comparisons address my main concerns. Therefore, I am raising my rating to 3, but this remains a borderline recommendation. Reactive Exploration still achieves higher mean returns in four of the six settings, and the Walker2D results leave the broader benefits unclear.

Dear Reviewer aq7C,

Thank you for raising your score and for your constructive feedback throughout! As Reactive Exploration is PPO-based, we now also added PPO-F to Table C.1 in the revision. It outperforms Reactive Exploration on InvertedPendulum and both MPLL settings, is worse on CartPole, and is on par on the others.

We agree with your assessment of the Walker2D results, which point to a clear direction for future work.

Reviewer_pvC9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes SsVD, which is a selective forgetting framework used for one-shot non-stationary reinforcement learning. NSRL is motivated by the drift of environmental dynamics. In the induced MDP, the reward function and the transition dynamics can change with time t, i.e., PtP_t and RtR_t. SsVD samples states from the state space, pulling their values toward a baseline. It adds one auxiliary loss.

Critical strengths
  1. This paper explored the interesting idea of a forgetting mechanism that is used to prevent stale value estimation in the training of Q-learning.

  2. They made an effort to improve the adaptability of RL algorithms to non-stationary environments, which is indeed a central problem of RL before its implementation in realistic environments.

Critical weaknesses
  1. In the experiment, the authors use MuJoCo environments MountainCar, CartPole, and InvertedPendulum, and the dynamic drifts are treated as parameter modifications. However, such model uncertainties can be captured by distributional robust frameworks in RL, and a comparison with DR-RL algorithms is needed.

  2. I am confused about the results reported in Figure 2, wherein cartpole-v1, mountaincar-v0, chooseBoxEnv-v0, and MultiPadlunarLander: is the performance of SAC-F missing? And it seems like SAC-F does not outperform other baselines in most of the environments.

  3. The reason why the forgetting mechanisms help with the adaptability of Q-learning was not explicitly explained. I do not understand how SsVD helps with the adaptability of Q-learning algorithms. It samples randomly from D_s, and then the forgetting mechanism helps discard outdated information for unvisited states. It can help prevent stale policies, but this does not provide a guarantee of a better/robust policy (in terms of returns) in an uncertain model.

Questions for authors
  1. SsVD changes reliance on old information with a chosen baseline b. Can you provide a baseline (either derived from theory or empirical) that will work well?

  2. Can you compare with other literature dealing with the non-stationary RL environment? For example, the meta-RL: https://arxiv.org/abs/1803.11347

2 comments

Dear Reviewer pvC9, we greatly appreciate your time spent to provide feedback! We thank you for recognizing the importance of our problem setting and the idea of using forgetting mechanisms to handle it.

Distributional / Meta-Rl

However, such model uncertainties can be captured by distributional robust frameworks in RL, and a comparison with DR-RL algorithms is needed.

Thank you for highlighting this important line of work! We include a comparison against R-contamination as distributional robust variant extended to the DQN/SAC setting. Our results show that R-contamination fails to provide robustness against the drifts considered (see Table C.1 in our revised document).

Can you compare with other literature dealing with the non-stationary RL environment? For example, the meta-RL: https://arxiv.org/abs/1803.11347

The setting of meta RL is different and does not apply to ours as it usually assumes a fixed set of environments, but we have a single environment possibly changing at every timestep. However, we follow the suggestion by Reviewer aq7C and benchmark against "Reactive Exploration" and "LILAC" which partly follow ideas from meta-RL (also in Table C.1).

Heuristic for baseline b

SsVD changes reliance on old information with a chosen baseline b. Can you provide a baseline (either derived from theory or empirical) that will work well?

We add this empirical experiment in Figure 4b: the baselines should be chosen around the midpoint of the state-value range of the given environment. This matches our theoretical hypothesis that choosing a neutral baselines is suitable.

Presentation

We improved the presentation at several places to improve readability.

I am confused about the results reported in Figure 2, wherein cartpole-v1, mountaincar-v0, chooseBoxEnv-v0, and MultiPadlunarLander: is the performance of SAC-F missing? And it seems like SAC-F does not outperform other baselines in most of the environments.

SAC is usually not used for discrete actions thus its omitted from the other environments. We improved the caption for Figure 2.

We want to be clear: as stated in Section 4.1, the rank differences are not statistically significant; the consistent gain over the base learner in all six environments, Figure 3 right, is our main claim.

The reason why the forgetting mechanisms help with the adaptability of Q-learning was not explicitly explained. I do not understand how SsVD helps with the adaptability of Q-learning algorithms. It samples randomly from D_s, and then the forgetting mechanism helps discard outdated information for unvisited states. It can help prevent stale policies, but this does not provide a guarantee of a better/robust policy (in terms of returns) in an uncertain model.

After a drift, the agent holds two kinds of wrong values: stale ones for states it no longer visits, and extrapolated, often overestimated ones for states it never visited. TD updates only correct visited states, so these errors persist and the greedy policy keeps acting on them. Figure 1 (top) shows exactly this for DQN.

Visited states get a TD update at every step, which outweighs the SsVD update: so only states that are not recently or often enough visited decay. The theory says the same in the tabular case: the error of a value not updated grows under drift - we bias this value thus to a baseline bb so it does not diverge arbitrarily. This however, does not provide a provable guarantee for a better policy.

Dear Reviewer pvC9,

Thank you again for your review. As suggested, we added a comparison to distributionally robust RL via R-contamination, which fails to provide robustness against the considered drifts, as well as to Reactive Exploration and LILAC, which partially build on ideas from meta-RL (Table C.1). We also added an empirical study on choosing the baseline b (Figure 4b), improved the presentation (including the caption of Figure 2), and clarified in our rebuttal how SsVD improves adaptability after a drift.

We would be grateful to know whether our responses resolve your concerns. If so, we would appreciate it if you would consider updating your assessment; if not, we would welcome the chance to address any remaining issues before the approaching deadline.

Best regards and many thanks for your time, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.