- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes VOSR, a replay-sampling module for off-policy safe reinforcement learning. It treats the replay distribution as a control variable and uses importance weighting to correct both replay-prioritization changes and, in the idealized analysis, state and action distribution mismatch. The authors derive a sampling distribution proportional to the baseline probability times a weighted magnitude of each transition's projected reward and cost-gradient contributions. They prove that this distribution minimizes a weighted sum of two scalar gradient-estimation variances, bound the probability of selecting an incorrect branch of a piecewise-linear safe optimizer, and argue that the ideal sampler preserves the optimizer's constrained stationary set. VOSR is integrated with SAC-Lagrangian, CRPO, and PCRPO and compared with uniform, TD-error, safety-oriented, and uncertainty-oriented replay on Safety Gymnasium tasks.
Critical strengths
-
The motivation is well targeted. Replay sampling is usually designed for reward or critic learning, even though safe policy optimizers depend on both reward and cost gradients. The proposed objective directly reflects this conflict and gives a principled explanation for why a transition with a large contribution to a safety-relevant gradient may deserve higher sampling probability.
-
The paper combines analysis with broad empirical testing. The importance-corrected estimator and the derivation (q^*(i)\propto b(i)\sqrt{(s_i^c)^2+\kappa(s_i^r)^2}) provide a meaningful baseline for designing safety-aware replay. Across three base optimizers and multiple Safety Gymnasium tasks, VOSR reports lower cost-gradient variance and lower training violation fractions while maintaining comparable returns. For an engineering-oriented replay method, these empirical results are important evidence.
Critical weaknesses
-
The theoretical guarantees do not directly characterize the implemented VOSR sampler. Theorem 1 derives the minimizer of the unconstrained weighted variance objective, (q^*(i)\propto b(i)\sqrt{(s_i^c)^2+\kappa(s_i^r)^2}). The practical algorithm instead uses (q_\epsilon(i)\propto b(i)\exp(\rho_i/\eta)), which is the solution of a different KL-constrained linear-priority objective, not the original variance objective with a KL constraint. In addition, the ideal unbiasedness argument assumes exact state-visitation and action ratios and untruncated importance weights, whereas the implementation estimates the state ratio and clips weights to [0.1, 10]. The priority scores are computed from noisy current-gradient estimates and only within a candidate pool. Consequently, the claims that the practical sampler is variance-optimal, leaves the expected optimizer update unchanged, and preserves the constrained stationary set should be stated as idealized claims unless the effects of these approximations are analyzed. In particular, unbiased gradients do not by themselves imply an unbiased update for branch-switching or projection-based nonlinear optimizers.
-
The method's computational and scalability costs are not evaluated, although they are central to the proposed replay framework. Standard prioritized replay can maintain TD-error priorities over a large buffer with a tree data structure, whereas VOSR's priority depends on the current actor score, both critics, global reward/cost directions, importance ratios, and a learned state-density ratio. These quantities change as the policy and critics are updated, so maintaining accurate priorities over the entire buffer would require repeatedly computing per-transition gradient projections. The implementation therefore scores only a candidate pool of 512 transitions before sampling the training minibatch. This reduces cost but introduces another sampling approximation and can miss high-value transitions elsewhere in a large buffer. The paper reports neither wall-clock or throughput overhead nor the effect of candidate-pool size, density-ratio refresh frequency, or fixed-compute comparisons. Moreover, the appendix changes effective safety margins and SAC multiplier settings between VOSR and some baselines, which makes it difficult to attribute improvements solely to replay sampling. These omissions are particularly consequential for an engineering contribution whose practical value depends on whether the safety gains justify the additional computation.
Questions for authors
-
Please clarify the exact status of the practical sampler relative to Theorem 1. What is the theoretical objective optimized by the exponential KL-tempered distribution, and how do state-ratio estimation error, weight clipping, priority estimation from a candidate pool, and nonlinear branch selection affect unbiasedness and stationary-set preservation? A comparison between the ideal (q^*), the KL-tempered sampler, and the implemented sampler would help delimit the guarantees.
-
What is the end-to-end computational cost of VOSR? Please report wall-clock time, update throughput, memory, and additional backward passes relative to each replay baseline, together with candidate-pool-size and density-ratio-refresh ablations. Under identical safety margins, multiplier hyperparameters, and a fixed compute budget, does VOSR retain its reward and violation improvements? These comparisons are needed to assess whether the method scales beyond the relatively small candidate pool used in the implementation.
3 comments
Thank you for the clarifications. I find the revised distinction between the idealized analysis and the implemented VOSR helpful, and I appreciate the additional analysis of the practical approximations. Some gap between the theoretical analysis and the implemented method remains. While the revisions clarify which results apply under the practical approximations, the claims should be carefully scoped to distinguish guarantees for the idealized sampler from those established for the deployed implementation. VOSR also incurs substantial runtime overhead, but given the reported improvements in return, cost, and constraint violations under a matched wall-clock budget, I consider this trade-off worthwhile. I have updated my rating to 3.
We agree that the guarantees for the idealized sampler must be kept separate from those for the deployed implementation. In the submitted revised version of the paper, Table 2 (in the Appendix/supplementary material) summarizes which claims hold in each setting, and the main text (Sec. 4.2, Theorem 3, and the Conclusion) maintains the same distinction, with the relevant text marked in blue.
Question 1: Status of the practical sampler relative to Theorem 1 and its guarantees
Response. The practical sampler is a smoothed approximation of the ideal sampler of Theorem 1, not its exact solution. Since the correction b(i)/q(i) in Eq. (3) keeps the gradients unbiased for every sampler (Lemma 1), the sampler can only change their variance. Theorem 1 minimizes this variance in Eq. (5) and gives the exact sampler q⋆(i)∝b(i)ζi. However, q⋆ can place most of its probability on a few high-score transitions, so each minibatch keeps repeating the same samples and ignores most of the replay buffer, which reduces replay diversity. Adding the KL constraint of Eq. (5) prevents this concentration, but then the problem has no closed-form solution. Proposition 2 therefore solves a simpler problem. It maximizes the expected score ∑iq(i)ζi subject to KL(q∥b)≤ϵ, which gives the exponential KL-tempered sampler qϵ⋆ of Eq. (6). This sampler still favours high-score transitions while staying close to uniform replay, but it needs exact scores, which depend on the true gradient directions and action values that are unknown during training. Therefore, the deployed VOSR uses the form of Eq. (6) with estimated scores. These scores are clipped to remove outliers and standardized within a candidate pool, so that their scale stays similar throughout training (Section 4.2). Table 8 shows that, relative to Uniform Replay, the cost-score variance is 0.32 for q⋆, 0.46 for qϵ⋆, and 0.52 for the deployed sampler, while the deployed sampler uses the most diverse set of transitions.
The approximations do not remove the guarantees that matter (Appendix A.1.5). Scoring a candidate pool adds variance but no bias (Lemma 2). Weight clipping and state-ratio error shift the mean gradients by the same small amount for every sampler (Proposition 3, Table 9), so all samplers keep one shared stationary set (Theorem 3). For nonlinear branch selection, Theorem 2 shows that wrong branches do not depend on the sampler, while wrong-direction steps grow with variance, which VOSR reduces (Table 10). Table 2 summarizes which claims hold for each sampler. For viewing Table 2, Table 8, Table 9 and Table 10, kindly visit the link https://anonymous.4open.science/r/variance-optimal-replay-0606/README.md. These tables are presented in Appendix A.1 (Table 2) and Appendix A.3 (Table 8, Table 9 and Table 10) of the paper.
Question 2: End-to-end computational cost and comparison under a fixed compute budget
Response. VOSR has a higher computational cost per step, but provides better safety and performance than baseline samplers. It takes 91–96% more wall-clock time per environment step than Uniform Replay and uses 0.34–0.37 GB more GPU memory than Uniform Replay (refer to Table 11, and Table 12). Its computational cost depends on the candidate-pool size rather than the replay buffer size, so increasing the buffer does not increase this overhead. Larger pool size of 1024 and 2048 increase runtime without improving performance over the default size of 512. Refreshing β every 250 steps gives the best results (Table 13). We also compare all samplers under the same wall-clock time and optimizer settings (Table 14). The baselines complete 4.6–5.9 million steps compared with 3 million for VOSR. Despite fewer steps, VOSR achieves a higher return (1262.4 versus 1052.4), lower cost (2.9 versus 7.5), and fewer training violations (0.025 versus 0.182). These results show that VOSR improves safety through better replay sampling that requires more computation. For viewing Table 11, Table 12, Table 13 and Table 14, kindly visit the link https://anonymous.4open.science/r/variance-optimal-replay-0606/README.md. These tables are presented in Appendix A.3 of the paper.