- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Space-sampled Value Decay (SsVD), which regularizes value estimates at sampled states toward a baseline to support non-stationary adaptation. It integrates the mechanism into DQN and SAC and provides a tabular theoretical analysis. Experiments on six non-stationary environments and three exploration tasks report improvements over the corresponding base algorithms.
Critical strengths
- The paper combines theoretical insights with empirical evaluation. The experimental results support the intuition behind value decay and provide evidence of its practical usefulness.
- The method adds a single auxiliary loss to the standard training objective. This makes it straightforward to implement.
Critical weaknesses
-
The method uses environment-specific values of and , but their selection procedure is insufficiently explained. Please clarify how these parameters should be chosen before deployment and provide sensitivity analysis.
-
The experiments do not establish a clear overall advantage over PPO, and the evidence for substantial gains on SAC is limited. Further analysis would help clarify when SsVD provides a practically meaningful improvement over these baselines.
-
The non-stationary evaluation focuses on relatively simple tasks and a limited set of drift patterns. Testing more challenging environments and more diverse reward and dynamics shifts would strengthen the generality of the conclusions.
I would be happy to raise my score if the authors can adequately address these three concerns.
Questions for authors
-
How should and be selected in practice—for example, for humanoid control in a continually changing environment with unknown future drift? In such a setting, what would motivate choosing SsVD over PPO directly?
-
How do the results change when the UTD ratio of DQN/SAC is increased? Do and require recalibration, and does SsVD retain its benefits when the base agents receive comparable UTD tuning?
-
Could a similar mechanism improve PPO, whose value estimates may also become outdated despite its on-policy updates? In particular, could an optimistic value prior improve performance, and how would it interact with advantage estimation and policy learning?
5 comments
Thank you for your response and the additional experiments, including Walker2D, PPO-F, and the sensitivity analyses.
I. Walker2D results: The improvement over SAC under slow drift is only about 1.7%, which is small relative to the substantial variability across seeds. Under fast drift, SAC-F performs worse on average. These results therefore do not establish a clear practical benefit over SAC in the newly added control task.
II. Hyperparameter selection under unknown drift: The midpoint heuristic for b and the sensitivity analyses are helpful. However, it remains unclear how to select b, η, and ξ before deployment when future drift is unknown. In particular, the paper does not clearly establish whether parameters selected independently of the test drift conditions retain their benefits across unseen drift speeds or patterns.
I appreciate the authors’ efforts, but these concerns remain insufficiently addressed. I am therefore inclined to maintain my rating of weak rejection.
Thank you for the careful reading and for engaging constructively throughout the discussion phase! We would like to clarify precisely what the new experiments establish regarding the scope of our claims and pre-deployment hyperparameter selection:
I. Role of Walker2D & Practical Benefit Over Base Algorithms
We agree that SAC-F matches rather than outperforms vanilla SAC on the periodic Walker2D task, and we intentionally present Section 4.4 ("Limits of scaling to higher-dimensional control") to transparently document the boundary of naive uniform sampling in unbounded 17D state spaces (Appendix B.4). Two points contextualize this result within the paper's broader findings:
- Core empirical claim on bounded spaces: Where valid state-space sampling is well-defined - our six primary non-stationary benchmarks -
SsVDconsistently improves over its direct base algorithm (DQNandSAC) in 6 out of 6 cases (Figure 3, right), including substantial gains forSAC-Fwhen drift permanently invalidates buffered experience. - The mechanism still transfers to 17D control: Even with our simple bounding-box sampling heuristic on
Walker2D, applyingSsVDwith the exact same parameters (no retuning) improvesPPOunder both drift speeds and on every single seed:1426 ± 182vs.925 ± 70(slow) and1417 ± 188vs.1251 ± 122(fast). Furthermore, restricting decay samples purely to the replay buffer degradesSAC-FonWalker2Djust as it does on five of the six primary benchmarks (Figure 4a), confirming that out-of-buffer value decay is a consistent principle worth extending with manifold-aware samplers in future work.
II. Selecting , , and Before Deployment Rather than tuning hyperparameters to test drift profiles, our evaluation demonstrates that fixed, pre-deployment defaults retain their benefits across unseen drift directions, speeds, and even the absence of drift:
- depends only on the known reward range and , not the drift: The neutral midpoint heuristic (, Figure 4b) requires only design-time reward bounds and - and in practice, even a simple default often sufficed without tuning (Table C.2).
- Wide defaults and insensitivity for and : Across all 7 benchmarks in Table C.2, we use only two coarse settings: and . On lower-dimensional tasks where the state space is directly sampled, performance is flat across and across an order-of-magnitude variation in (Figure C.3a–b).
- No-retuning transfer across unseen drift regimes (including zero and reversed drift):
- Stochastic trajectories: Probabilistic schedulers generate distinct drift trajectories per seed under fixed (Figure B.2).
- Unseen "no-drift" deployment: Keeping completely unchanged when drift is removed entirely (stationary ablation, Figure B.5) matches or improves over vanilla
DQN/SAC; clearly showing no penalty if anticipated drift does not occur. - New experiment: Unseen reversed (increasing) drift: To test transfer to an unseen drift pattern directly, we evaluated
CartPoleandInvertedPendulumunder increasing parameter drift (reversing the decreasing drift in the main paper) while keeping , , and strictly identical to Table C.2 (mean online return ± sample std over 10 seeds):
| Environment | Drift | Limited | Base | SsVD |
|---|---|---|---|---|
| CartPole (DQN) | reversed (force 10 → 50) | 72 ± 99 | 83 ± 10 | 288 ± 25 |
| CartPole (DQN) | paper (force 10 → 0.9) | 49 ± 15 | 99 ± 68 | 394 ± 12 |
| InvertedPendulum (SAC) | reversed (cart mass 10.5 → 31.4) | 950 ± 101 | 910 ± 14 | 984 ± 4 |
| InvertedPendulum (SAC) | paper (cart mass 10.5 → 0.5) | 390 ± 92 | 945 ± 13 | 985 ± 3 |
Under both held-out increasing drifts - with all SsVD settings unchanged - SsVD remains clearly best. Note that on InvertedPendulum, increasing cart mass is milder on the initial policy than decreasing it (a policy frozen at 100k steps still averages 950 ± 101), yet continued off-policy training under shifting mass degrades vanilla SAC to 910 ± 14 whereas SsVD stabilizes learning at near-optimal return (984 ± 4) with minimal variance.
Together, these results show that for bounded observation spaces, practitioners can select a priori from reward bounds and standard coarse defaults, retaining strong performance across decreasing, increasing, and stationary regimes. In high-dimensional unbounded spaces (Walker2D), sensitivity increases alongside the state-sampling approximation, which we highlight transparently in Section 4.4 and Section 6 as an avenue for future adaptive sampling methods.
Dear Reviewer AfsL, we highly appreciate your input and openness to participate in the rebuttal process! We thank you for highlighting how our experiments confirm the intuition behind our proposed mechanism.
Scaling to higher dim. control
We extend our existing set of different drift patterns (ChooseBox and MultiPadLunarLander have changing rewards) by utilizing two different sine based periodic drifting characteristic for the higher-dimensional Walker2D environment. For this we adapt the original sampling scheme of SsVD (Appendix B.4).
This is now a new section 4.4 in our revised document. Table 2 clearly shows that PPO fails and LILAC falls behind considerably in the considered problem setting. SsVD is competitive for slow drift but learns more slowly in case of fast periodic drift. Following your suggestion we also included PPO-F (see below).
We improved the already existing discussion of this SsVD's limitation: we leave further modifications for higher dimensional state-spaces and image based observations as future work.
To test whether the mechanism carries over to on-policy methods, we implemented PPO-F. It adds the SsVD term to PPO's state-value critic.
On Walker2d, PPO-F improves PPO's mean online return by factors 1.1x/1.5x depending on the chosen drift speed, so SsVD can be helpful for PPO. On InvertedPendulum, however it does not fix the case where PPO fails to track the optimal policy late in the experiment. We added PPO-F to Table 2 and leave further analysis for on-policy algorithms to future work.
Sensitivity to hyper-parameters
For the baseline b see our answer to Reviewer pvC9 or Figure 4 b) (tldr: choose the midpoint). Figure C.3 a) + b) shows a sensitivity analysis with respect to η and ξ. We find SsVD to be robust to these choices in most tested cases except for MountainCar (too high values for η degrade performance) and on Walker2D, thus higher-dimensional tasks may require tuning η and ξ.
This ablation is adressed in Figure 4(c): for the selected environments we find that UTD to not have a significant effect - it neither improves the base algorithms nor does it hurt SsVD. Only for the newly introduced Walker2D we observe SAC improvement for a higher UTD that we also, but only for even higher UTD, observe for SsVD - we ultimately believe this to be environment dependent.
Recalibrating η with the UTD ratio (dividing it by the UTD factor) did not help on any environment, so we keep the parameters fixed.