acceptodds
Under review as a conference paper at ICLR 2027

Episodic Control for Safe RL with Guarantees

Abstract

On-policy safe reinforcement learning (RL) methods provide strong stability and constraint guarantees but are sample-inefficient, while off-policy methods improve data efficiency at the cost of stronger assumptions to obtain guarantees, creating a fundamental tension in safety-critical settings. We introduce the Safe Integration Framework (SIF), a class of admissible critic corrections that inject bias into on-policy value estimation while preserving contraction of the Bellman operator and bounding deviation from the nominal fixed point. We further show that SIF preserves constrained trust-region optimization guarantees up to an explicit slack term, and yields primal-dual convergence to a bounded neighborhood of the nominal saddle point. As an instantiation, we propose Safe Proximity-guided Episodic Control (SPEC), a lightweight retrieval-based plugin that augments on-policy cost critics with uncertainty-aware corrections. Across six maximum-difficulty Safety-Gymnasium navigation tasks spanning the Button, Goal, and Circle families, SPEC applied to TRPO-PID yields a more consistent constraint profile than all baselines while avoiding reward collapse.

Primary area:
reinforcement learning
Keywords:
safe reinforcement learning, episodic control, on-policy reinforcement learning, trust-region methods, Lagrangian methods, off-policy information, theoretical guarantees, retrieval-based reinforcement learning, Safety-Gymnasium

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KoU9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how episodic information can augment on-policy safe reinforcement learning while controlling the resulting estimation bias. Its Safe Integration Framework (SIF) adds a clipped, Lipschitz correction to the lambda-return Bellman operator. The analysis establishes contraction and a fixed-point deviation bound, then propagates critic error into trust-region performance bounds and a local robustness result for primal–dual dynamics. The practical instantiation, Safe Proximity-guided Episodic Control (SPEC), retrieves historical trajectories in a learned cost-critic embedding, forms neighbor-based multi-step cost estimates, and uses their residuals to adjust critic targets. Corrections are weighted by IVF-cell concentration and annealed over training; the method also stacks environment observations. Experiments apply SPEC to TRPO-PID on six Safety-Gymnasium tasks with five seeds, comparing against safe-RL baselines and SafeEC, with ablations of correction strength, weighting, observation history, and encoder warmup.

Critical strengths
  1. SIF makes the correction magnitude and sensitivity explicit rather than treating retrieved returns as reliable on-policy targets. Theorem 3.1 and the subsequent perturbation analysis provide a transparent way to reason about the cost of introducing auxiliary information. This is useful because stale or mismatched episodic information can otherwise introduce unaccounted-for bias into safe policy optimization.

  2. SPEC is evaluated against both its base algorithm and a prior safe episodic-control method across six tasks. The beta-max-zero control retains environment observation stacking (Table 2), helping distinguish retrieval corrections from temporal-context changes. Table 1 shows potentially useful gains: PointButton cost falls from 32.3 to 18.1 relative to TRPO-PID at the same reported reward, and RacecarCircle reward is 14.8 versus SafeEC's 0.3 while SPEC's cost IQM remains below 25. These results support further investigation, even though they do not establish uniform superiority.

Critical weaknesses
  1. The necessity and theoretical advance of the proposed approach over existing off-policy policy evaluation (OPE) methods are unclear. Appendix A mainly discusses safe RL and episodic control, without discussing directly relevant estimators such as Retrace and V-trace or explaining why adapting existing OPE methods to the cost critic would be insufficient. Such methods do not automatically solve constrained policy optimization, but they address the data-reuse problem motivating SPEC. SPEC instead adds a clipped historical-return residual to an on-policy target, accepting both policy-mismatch and state-neighborhood bias. This is not intrinsically invalid, but clipping does not establish more accurate estimates, a better bias–variance trade-off, or protection against cost underestimation. Theorems D.2 and 3.1 show that a sufficiently small Lipschitz perturbation preserves contraction to a generally biased fixed point; they do not establish that retrieval improves evaluation, and would also apply to uninformative corrections satisfying the same assumptions. The numerical guarantees appear weak: Table 6 parameters give worst-case value and GAE error bounds of approximately 110.7 and 3702.3 using beta_max and L_delta = 1, while Remark E.6 acknowledges potentially vacuous continuous-state performance bounds. Moreover, exact-operator convergence and conditional local ODE stability do not establish convergence of the implemented neural SPEC–TRPO-PID updates. Appendix E, Eq. (14), also applies the nominal performance inequality to a biased critic before accounting for baseline error, a step requiring justification. These issues leave the claimed benefit over existing evaluation methods and standard perturbation analysis insufficiently substantiated.

  2. The empirical results do not convincingly establish improved safety consistency or a retrieval-specific advantage. Table 1 classifies methods as safe using cost IQM <= 25 across five seeds, yet SPEC's costs are 24.9 +/- 16.9 on CarGoal and 23.8 +/- 13.3 on RacecarCircle. Without per-seed feasibility statistics, cumulative training excess cost, and uncertainty estimates, these results provide limited evidence of reliable constraint satisfaction. The reward–cost trade-offs are also mixed: SPEC has lower reward and higher cost than TRPO-PID on DoggoCircle, and TRPO-Lag achieves higher reward and lower cost than SPEC on RacecarCircle. Furthermore, the beta-max-zero control already achieves RacecarCircle reward 14.1 and cost 15.0, versus 14.8 and 23.8 for full SPEC (Table 2). The absence of matched OPE, uniform-memory, or simple critic-regularization controls makes it difficult to attribute the gains to informative episodic retrieval rather than observation stacking or generic critic perturbation.

Questions for authors
  1. What prevents existing OPE estimators from addressing the historical-data reuse component, and what does SPEC gain by accepting uncorrected bias? Please clarify whether the analysis establishes anything beyond stability under bounded perturbations, whether a quantitatively useful safety or estimation bound is available, and which guarantees apply to practical SPEC–TRPO-PID. Please also justify Eq. (14): in a one-state zero-reward MDP with identical policies and delta = 0, an admissible constant critic -b gives positive GAE and makes that intermediate inequality assert 0 >= b. Can the argument be repaired by starting from the nominal inequality and explicitly propagating the critic error?

  2. Can you specify the evaluation window and aggregation procedure, and provide per-seed feasibility, cumulative training excess cost, and uncertainty estimates to substantiate improved safety consistency? Under matched architecture, observations, and training budget, how does SPEC compare with an appropriately adapted OPE estimator and a uniform-memory or simple critic-regularization control? These comparisons would help establish whether retrieval itself explains the reported gains.

3 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer KoU9

We thank the reviewer for their feedback but respectfully disagree with several interpretations of our contributions.

W1/Q1: Historical reuse and OPE. Retrace/V-trace can reuse historical costs under their assumptions, but do not themselves establish constrained policy guarantees. SPEC augments on-policy targets with bounded neighborhood estimates, sharing historical cost observations without behavior-policy ratios. Policy-mismatch and neighborhood bias remain, but that is the point of SIF, which asserts it's admissible. Clipping does not guarantee better accuracy, variance, or conservative costs, which is the point of SPEC. Our matched FQE comparison tests whether reuse alone explains SPEC's gains.

Contribution. Theorems D.2/3.1 establish contraction and bounded fixed-point deviation without requiring informative retrieval; they do not claim retrieval improves evaluation. This generality is intentional: SIF specifies conditions under which auxiliary information can be integrated while preserving conditional guarantees. Its scope also extends beyond critic stability: Theorem 3.2/Corollary E.4 bound reward/cost performance with explicit slack, while Appendix F analyzes local primal–dual dynamics. Retrieval's utility is evaluated empirically.

Bounds and practical scope. Our analysis deliberately prioritizes general admissibility and explicit control of perturbation effects. The submitted worst-case critic/GAE bounds are conservative: 110.7/3702.3 at β=0.15\beta=0.15, remaining nonzero at β=0\beta=0. Exploiting clipping and infinite-horizon GAE telescoping tightens these to 11.9/21.8 in the idealized setting, but doesn't certify safety. Table 2 provides evidence: βmax⁡=0\beta_{\max}=0 meets the cost-IQM budget on three tasks, versus all six for SPEC. Thus, these guarantees concern idealized operators and conditional local dynamics, not convergence of neural actor–critic training, which is the purpose of SPEC.

Eq. (14). The counterexample highlights the need to account explicitly for baseline error. We will clarify the argument by starting from nominal Eq. (2) and applying Lemma E.2 to propagate critic error, preserving the conditional reward bound under its assumptions. We will also clarify cost-corollary GAE accumulation and associated slack; see our response to 5ND4 Q2.

W2/Q2: Safety consistency and evaluation. We report five seeds over 2M steps. Feasibility counts seeds with final cost ≤25\le25. At 5k-step checkpoints, violation frequency is the fraction with cost >25>25; severity is mean excess C(t)−25C(t)-25 during violations. Both are seed IQMs. R/C SE denotes across-seed reward/cost standard error. Entries are TRPO-P →\to SPEC; near-saturated Circle tasks are omitted for space.

TaskFeasible seedsViol. freq.Avg. severityR/C SE
DoggoButton3/5 → 3/50.78 → 0.6015.9 → 8.60.02/3.1 → 0.07/2.2
PointButton1/5 → 4/50.35 → 0.4321.5 → 19.00.41/5.2 → 0.46/2.1
PointGoal2/5 → 3/50.49 → 0.4917.5 → 17.20.94/6.9 → 0.14/2.0
CarGoal0/5 → 2/50.72 → 0.6616.8 → 23.70.17/4.0 → 0.11/6.1

These results strengthen SPEC's safety case beyond aggregate IQM: final feasibility improves on three tasks and ties on one; violation frequency improves/ties on three, and severity improves on three. PointButton trades higher frequency for lower severity; CarGoal trades higher severity for fewer violations and greater feasibility. Reward SE falls on both Goal tasks, and cost SE falls except on CarGoal, where feasibility improves from 0/5 to 2/5.

Scope and task-level trade-offs. We disagree that mixed task-level outcomes negate SPEC's broader safety gains. Sections 4/6 acknowledge reward–cost trade-offs; uniform dominance is not claimed. The βmax⁡=0\beta_{\max}=0 control retains SPEC's encoders and observation history but meets the budget on only three of six tasks. Its stronger RacecarCircle result does not outweigh SPEC's broader feasibility. The per-seed results above further support these safety benefits despite task-dependent trade-offs.

Attribution and matched controls. Our response to n5bM provides stacking, random-memory, and critic-regularization controls. These reinforce SPEC's contribution.

We additionally evaluate FQE–TRPO-P, fitting its cost critic via FQE on a uniform 500k-transition buffer, matching observation history, hidden layers, and 2M-step budget. Its reward/cost IQMs (five seeds) are DoggoButton 0.2/16.5, PointButton −1.3/31.0, PointGoal −0.4/28.6, CarGoal −1.6/25.4. SPEC meets the cost budget on all four, versus one for FQE–TRPO-P, with comparable/higher reward on three. This supports SPEC's integration mechanism beyond generic historical reuse, though isolating the causal role of theoretical guarantees requires further study.

SIF's conditional guarantees and SPEC's empirical gains provide an advance beyond standard perturbation analysis and generic off-policy reuse.

Thank you for the rebuttal. However, my main concern remains.

Off-policy policy-evaluation methods have established theoretical guarantees. Retrace and V-trace are two examples: their analyses establish contraction and characterize the value functions their estimators converge to under stated assumptions (Munos et al., 2016; Espeholt et al., 2018). Many other OPE methods also provide theoretical guarantees. Although these methods are often presented for reward critics, their evaluation framework can also be applied to cost critics: replacing the scalar reward with cost preserves the Bellman form, subject to the relevant assumptions.

Although these methods are commonly presented for reward critics, their evaluation operators have the same form for a cost critic: the scalar reward signal is replaced by the scalar cost signal. Thus, provided the relevant assumptions hold for the cost signal and the data, the corresponding policy-evaluation guarantees apply to cost estimation as well. If those assumptions cannot be met in the paper’s setting, the authors should explain which ones fail and why. The paper and the rebuttal have not provided this analysis. It does not sufficiently discuss existing OPE methods, identify assumptions that make them unsuitable for this setting, or explain why their limitations motivate SPEC. Nor does it establish what advantages SPEC offers over these alternatives in estimation accuracy or practical applicability.

We thank the reviewer for raising this point. Since the beginning of the discussion phase, we have been working on the theoretical and empirical analysis of Retrace and V-trace within TRPO-PID to clarify their relationship to SPEC. Both analyses are now complete, and we would be happy to share the results with the reviewers.

Adaptation to TRPO-PID. Replay trains only the cost critic; the reward critic retains on-policy training, the actor uses fresh rollouts, and PID uses observed episode costs. During evaluation, we freeze the reference policy πk\pi_k and critic snapshot. Let wt=πk(at∣st)μ(at∣st),gt=γ(1−terminalt).w_t=\frac{\pi_k(a_t\mid s_t)}{\mu(a_t\mid s_t)},\quad g_t=\gamma(1-\mathrm{terminal}t). V-trace uses ρt=min⁡(ρˉ,wt)\rho_t=\min(\bar\rho,w_t), ztV=λtrmin⁡(zˉ,wt)z_t^V=\lambda{\mathrm{tr}}\min(\bar z,w_t) and

\delta_t^V&=c_t+g_tV_c^-(s_{t+1})-V_c^-(s_t),\ e_t^V&=\rho_t\delta_t^V+g_tz_t^Ve_{t+1}^V,\quad Y_t^V=V_c^-(s_t)+e_t^V. \end{aligned}$$ Retrace uses $z_t^Q=\lambda_{\mathrm{tr}}\min(1,w_t)$, $\bar V_c^-(s)=\mathbb E_{a\sim\pi_k}Q_c^-(s,a)$ and $$\begin{aligned} \delta_t^Q&=c_t+g_t\bar V_c^-(s_{t+1})-Q_c^-(s_t,a_t),\ e_t^Q&=\delta_t^Q+g_tz_{t+1}^Qe_{t+1}^Q,\quad Y_t^Q=Q_c^-(s_t,a_t)+e_t^Q. \end{aligned}$$ Recursions stop at segment boundaries; true terminations suppress bootstrapping, while time-limit truncations retain it. We fit the critic to detached targets. Fresh-rollout cost GAE uses $V_{c,\phi}$ for V-trace and $\mathbb E_{a\sim\pi_k}Q_{c,\phi}$ for Retrace, whose action expectations use Monte Carlo samples. The actor retains $\widehat A_\nu=(\widehat A_r-\nu_k\widehat A_c)/(1+\nu_k)$. **Theoretical analysis.** Appendix G of the revised manuscript contains the new analysis. We also incorporated Reviewer 5ND4's clarifications, applying Lemma E.2 directly to the nominal bound instead of the Eq. (14) step. We can share this revision. However, incorporating all additional experiments material will require more time. The trace evaluation operators are contractions under their original assumptions. Our extension connects their errors to SIF's bounds: given a valid nominal bound and uniform control of cost-critic error relative to the reference policy, the GAE perturbation and conditional trust-region performance arguments apply. Local Lagrangian robustness additionally requires the paper's idealized mean primal–dual ODE assumptions and sufficiently small critic-induced drift perturbations, uniformly in time. Retrace introduces no intrinsic clipped-target mismatch; its action expectation must be integrated, which adds integration error. V-trace requires accounting for its clipped-policy target or the actual mixed-replay fixed point. These guarantees extend beyond SPEC but remain conditional, without end-to-end guarantees for the implementations. We thank the reviewer for suggesting this analysis, which strengthens the paper. We now distinguish two components of SIF. First, its conditional policy-performance and local-robustness guarantees depend on a bound on cost-critic error relative to the reference policy and therefore extend to Retrace and V-trace under the required assumptions (Appendix G). Second, its fixed-point perturbation analysis addresses SPEC-type bounded additive corrections, a distinct mechanism from the Retrace and V-trace operators. This component provides an explicit worst-case deviation in terms of correction strength and cap. While this two-component distinction is not yet explicitly presented in the revised manuscript, we will incorporate it into the final version. Our evidence for SPEC’s practical advantage comes from the empirical comparison. **Empirical comparison.** Entries report reward IQM / cost IQM across five seeds. **Bold** indicates cost IQM ≤25. | Env | Retrace | V-trace | SPEC | |:---|---:|---:|---:| | DoggoButton | 0.27 / 33.6 | 0.30 / 36.0 | 0.17 / **24.1** | | PointButton | −1.45 / **20.1** | −1.19 / **22.1** | −1.29 / **18.1** | | PointGoal | −1.46 / 31.0 | −0.78 / 43.6 | −1.17 / **24.4** | | CarGoal | −0.54 / 27.5 | −0.94 / 32.3 | −1.49 / **24.9** | | RacecarCircle | 8.63 / 25.6 | 6.40 / **15.6** | 14.81 / **23.8** | | DoggoCircle | 2.37 / **21.7** | 2.10 / **22.1** | 2.48 / **18.3** | **SPEC achieves cost IQM ≤25 on all six tasks, versus two for TRPO-PID with a Retrace critic and three with a V-trace critic.** We did not directly evaluate critic accuracy because exact reference values are unavailable, and training MSE against noisy or bootstrapped targets alone does not establish true value-function accuracy. We will incorporate the full empirical comparison alongside the new theoretical extension in the revision. This will strengthen the paper through an additional theoretical contribution extending SIF’s conditional guarantees to Retrace and V-trace critics, establishing SIF’s broader applicability, and empirical comparisons reinforcing SPEC’s superior constraint-satisfaction profile.
Reviewer_5ND4OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the Safe Integration Framework (SIF) to incorporate off-policy information into on-policy safe RL with bounded critic bias and theoretical guarantees. Building on SIF, the authors introduce SPEC, which uses episodic memory and adaptive corrections to improve cost estimation. The bias in off-policy experience is controlled with theoretically analyasis. Experiments on 6 Safety-Gymnasium tasks show more consistent constraint satisfaction than baselines while maintaining competitive rewards.

Critical strengths
  1. Good motivation and reasonable method to integrate off-policy meomry into on-policy safe RL. The paper proposes a general framework for combining the sample efficiency advantage of off-policy RL with on-policy safe/contrained RL through bounded critic corrections. The Bellman contraction and fixed-point deviation analyses provide a theoretical foundation for controlling the bias of transfering experience/information.

  2. Practical algorithm with competitive results. The proposed SPEC can be integrated into existing on-policy safe RL algorithms without modifying their policy optimization procedures. This compatability is useful for practial applications. Experiments across 6 Safety-Gymnasium tasks demonstrate consistent constraint satisfaction while avoiding the severe reward collapse observed in other episodic-control methods.

Critical weaknesses
  1. Issues in the theoretical derivation. The proofs and the formulation of the key theorems and corollaries are sloppy, with quite some inconsistencies. Details are given in the comments. These issues undermine the claimed policy performance and safety guarantees.

  2. Indirect evidence about the claimed methodological benefits. Although SPEC achieves cost performance below the safety threshold, several results exhibit substantial variance, and reward improvements are inconsistent. An important missing part is that the paper provides no direct quantitative evaluation of sample efficiency improvements, which is the first motivation of the study.

Questions for authors

Q1. The math formulas seem sloppy, and some rescaling factors seem missing. That causes confusion for readers. For example, Jr(π)J_r(\pi) uses the normalized discounted state-visitation distribution, but Eq.(2) uses unnormalized discounted returns. Similar for the inconsistency between the episode-level threshold and the limit on the cost values. Also, in the proof of Corollary 3.3, the discounting factor 1/(1−γλ)1/(1-\gamma\lambda) also seems to be missing in the GAE formula. Please run through the formulas carefully and correct these inconsistencies. Correct me if I understand anything wrong.

Q2. I do not get Eq.(14). Why we can get Eq.(14) from Eq.(13)? It seems that the bias in Eq.(14) is ignored? If I understand correctly, the bias is controlled by Lemma E.2, right? Correct me if I am wrong. As Theorem 3.2 is important, please explain the proof better. Directly transferring Eq.(13) on the true value to Eq.(14) on the SIF-based value/advntage is confusing.

Q3. SPEC uses entropy to measure retrieval uncertainty. However, retrieved trajectories originate from potentially different policies, introducing distribution mismatch. Could the authors explain why the proposed uncertainty measure can effectively represent retrieval accuracy? Why not choose other metrics? What is the motivation here?

Q4. Improving sample efficiency of on-policy constrained RL is the first motivation of the paper. However, the experiments report reward and cost under a fixed training budget. Could the authors provide direct, numerical metrics for sample-efficiency comparisons? With cost constraints, how to measure the so-called sample efficiency? How fast does the cost converge below the threshold? Or how fast does the reward increase with env steps? I believe that there is a trade-off relationship between them. Please explain and provide such results.

The motivation and the method design themselves are reasonable, but the theoretical part is a little bit sloppy and redundant, and perhaps not all of the theorems/corollaries are necessary.

4 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer 5ND4

We thank the reviewer for recognizing the strong motivation and empirical results of our work. We clarify the theoretical concerns and provide additional evidence below.

Weaknesses. (1) Theory. We will standardize returns, restore the corollary’s geometric-series factor, and clarify Eq. (14) via Lemma E.2 (Q1/Q2). Contraction and fixed-point results remain intact, while conditional performance and cost-violation bounds retain their structure with revised slack. Thus, SIF still bounds the effects of critic corrections on performance and safety under the stated assumptions. (2) Evidence. We address variability in our response to n5bM; and we note that SIF targets average safety gains, not uniform dominance. Our analysis (Q4) shows earlier median joint-target time on all six tasks (marginally on CarGoal).

Q1. Thank you for highlighting these points. With normalized discounted visitation, Edπ,π[x]=(1−γ)Eρ0,π∑t≥0γtxt.\mathbb E_{d^\pi,\pi}[x]=(1-\gamma)\mathbb E_{\rho_0,\pi}\sum_{t\ge0}\gamma^t x_t.

We will align the definition of JJ with the unnormalized discounted-return convention used in Eqs. (1)–(2), and express the associated thresholds consistently. We will also explicitly distinguish the theoretical discounted-cost constraint from the experimental limit of 25 on undiscounted episode cost, since these quantities are not generally related by a constant rescaling.

For Corollary 3.3, the GAE definition remains the standard discounted sum. The factor 1/(1−γλ)1/(1-\gamma\lambda) enters when bounding that sum: if ϵβ\epsilon_\beta uniformly bounds the expected absolute one-step TD residual along the continuation trajectory, then E[A^λ,β]≤ϵβ1−γλ.\mathbb E[\widehat A_{\lambda,\beta}]\le \frac{\epsilon_\beta}{1-\gamma\lambda}. We will retain this factor in the proof, specify the continuation law, and revise the resulting quantitative slack.

Q2. We will clarify how the perturbation bounds enter the performance analysis: Eq. (13) controls one-step TD-residual inflation, while Lemma E.2 bounds the advantage bias. Let C,ϵrC,\epsilon_r be as in Eq. (2), E≥∥Vβ−Vπ0∥∞E\ge\lVert V_\beta-V^{\pi_0}\rVert_\infty, B=(1+γ)E/(1−γλ)B=(1+\gamma)E/(1-\gamma\lambda), and D=(1+γ)ED=(1+\gamma)E, where VβV_\beta is the reference-policy SIF fixed point. Under the assumptions and consistent return and trajectory conventions required for the nominal Eq. (2), applying Lemma E.2 directly gives

\ge \frac{\mathbb E A_\lambda-C\epsilon_r}{1-\tilde\gamma} \ge \frac{\mathbb E A_{\lambda,\beta}-B-C\epsilon_r}{1-\tilde\gamma} \ge \frac{\mathbb E A_{\lambda,\beta}-B-C(\epsilon_r+D)}{1-\tilde\gamma}.$$ Taking $E=\eta/[1-(\tilde\gamma+\beta L_\Delta)]$, as permitted by Theorem 3.1 and clipping, recovers the algebraic form of Theorem 3.2. The additional penalty $CD\ge0$ only weakens the lower bound. We will state and justify the continuation law, visitation distribution, and KL convention under the return convention from Q1, revising the nominal coefficient where necessary. **Q3.** SPEC has no designated ground-truth retrieval target to define accuracy, so IVF-cell entropy serves as a cheap proxy for uncertainty, downweighting heterogeneous neighborhoods using existing assignments (Section 4). The rolling 500k-transition memory favors recent policies, while short returns bootstrapped from the current cost critic mitigate policy mismatch. However, SIF does not require unbiased retrieval: its guarantees accommodate policy-dependent bias, which is a key contribution. **Q4.** Q4. We track reward and cost during training, beyond fixed-budget endpoints. Joint-target time is the first step when return reaches the baseline's final $R^*$ and cost ≤25. Parentheses show successful seeds out of 5. Empty cells have undefined slopes. |Task|Joint-target (M steps)|Steps to ≤25 (M)|Reward slope pre-≤25| |:---|:---:|:---:|:---:| |DoggoButton|0.18 (5/5) → **0.12** (5/5)|0.01 → 0.03|| |PointButton|>2 (2/5) → **1.19** (4/5)|0.27 → 0.58|−8.05 → −3.23| |PointGoal|0.59 (3/5) → **0.48** (5/5)|0.37 → 0.44|−4.38 → −3.17| |CarGoal|0.67 (5/5) → **0.66** (4/5)|0.51 → 0.22|−0.05 → −1.16| |RacecarCircle|1.72 (3/5) → **0.89** (3/5)|0.12 → 0.15|0.01 → 1.19| |DoggoCircle|>2 (2/5) → **1.51** (4/5)|0.01 → 0.01|| We prioritize joint-target time, measuring when reward and safety targets are both met. SPEC improves it on all six tasks. On CarGoal, despite fewer successful seeds (4/5 vs. 5/5), SPEC reaches the target earlier and meets the final IQM cost constraint, unlike TRPO-P. We also report steps to first cost ≤25 and preceding reward slope. SPEC generally reaches the cost threshold later but preserves reward better, suggesting a gradual reward–safety trade-off. CarGoal shows the reverse: faster cost reduction but greater reward loss. SPEC’s cost-focused correction thus shifts the reward–safety balance across tasks without uniformly accelerating cost convergence. SIF's guarantees and SPEC's sample-efficiency gains substantiate our contributions. We will streamline any theoretical results going forward.

Thank you for your explanations. I have one follow-up question:

  1. In your response to Q4, Joint-target time seems like a reasonable sample efficiency metric. But does "baseline's final R∗R^*" refer to vanilla TRPO-PID, the corresponding baseline for each method, or the best-performing baseline in each environment? Also, is the baseline's final reward threshold computed across seeds or separately for each seed? These two choices can be very different, right? Could you clarify this if the table will be included in the manuscript?

I do not have any other questions for now. If the authors upload a revised manuscript during the discussion period, I would like to check some of the revised theoretical part again.

We thank the reviewer for the follow-up and answer the question directly.

To clarify, the baseline refers specifically to vanilla TRPO-PID, and the final reward threshold is TRPO-PID's final reward IQM across five seeds, shared across all methods within each environment. Thus, our joint-target metric measures how quickly each method achieves the same reward level as vanilla TRPO-PID while simultaneously satisfying the safety constraint, rather than using method-specific or seed-specific reward targets. This provides a consistent reference for comparing sample efficiency across methods.

All tables and discussions that we have provided here during the discussion period will be included in the revised manuscript.

For now, we have uploaded a revised manuscript on incorporating the theoretical streamlining and revisions, which are available for review. The revision also extends SIF’s conditional policy-performance and local-robustness analysis to Retrace and V-trace cost critics under the required assumptions, based on feedback from KoU9.

Incorporating all the additional experimental tables will require more time, but we plan to include them in the final revision.

Thank you for your clarification. I do not have any more comments and will maintain my positive rating. Thank you again for this interesting study.

Reviewer_n5bMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper aims to make on-policy safe RL more data-efficient without losing its theoretical guarantees.

  • SIF (Safe Integration Framework). Adds a scaled, clipped and Lipschitz correction to the critic’s training target. The authors show that the corrected critic still contracts to a fixed point within an explicit distance of the true value. Trust-region performance and cost bounds then hold with an added slack term, and Lagrangian primal–dual dynamics converge locally to a neighbourhood of the nominal saddle point.

  • SPEC. An instantiation of SIF that corrects only the cost critic. It uses n-step returns from nearest-neighbour trajectories retrieved in a learned embedding space. The correction is weighted by how concentrated the retrieved neighbours are and is annealed to zero over training. The method also stacks the history of environment observations.

  • Validation. SPEC is added to TRPO-PID and evaluated on six Level-2 Safety-Gymnasium tasks with 5 seeds. It is the only method whose IQM cost stays within the limit on all six tasks, with reward comparable to TRPO-PID. Ablations cover β_max, adaptive versus fixed β, observation stacking and encoder warmup.

Critical strengths

The plug-in is simple and could be added to existing on-policy methods. The theoretical results are stated with their assumptions and limitations acknowledged, and the ablations are reasonably thorough.

Critical weaknesses
  1. The gains are not attributed to retrieval. It is unclear whether the baselines use observation stacking. The β_max = 0 results suggest that much of the gain comes from stacking rather than from the episodic correction. The cost margins are also small relative to seed variance.

  2. The motivation is unclear and untested. Why injecting bias into the critic is desirable is never argued, for instance as variance reduction under sparse costs. No advantage over alternatives such as ensembles is shown, and sample efficiency, the stated motivation, is not measured.

  3. The theory is modest. The results follow standard perturbation arguments. With the reported hyperparameters the bounds are numerically loose, and their relevance to the implemented algorithm is not made clear.

  4. The scope is narrow.

  • The method is evaluated with only one base algorithm and on cost critics only.
  • Baseline coverage is limited: no further SafePO methods and no off-policy methods.
  • The claim of “avoiding reward collapse” sits uneasily with the predominantly negative rewards.
Questions for authors

The paper should report the computational overhead of SPEC, including the cost of maintaining the retrieval structure and querying it during training. This is particularly important given the emphasis on sample efficiency: improvements in environment interaction efficiency should be considered alongside the additional computational cost.

3 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer n5bM

We thank the reviewer but believe several concerns reflect misunderstandings addressed below.

W1/W2. Isolating retrieval is important and we add those baselines here. The table compares: TRPO-P+envstack, TRPO-P+envstack+critic-L2=.1.1, random retrieval with SPEC's architecture, ensTRPO-P (TRPO-P+envstack with an ensemble of 5 cost critics), and the off-policy DDPG-P. For TRPO-P and SPEC, please refer to Table 1 of the paper.

Env+envstack+L2 .1ensTRPO-PRandomDDPG-P
DoggoButton0.2/27.80.0/29.60.1/28.40.2/21.10.00/0.0
PointButton−1.5/24.4−1.7/21.1−0.6/26.3−1.2/26.5−0.77/61.5
PointGoal−1.4/31.0−2.6/26.1−0.7/28.6−0.7/32.1−0.17/145.0
CarGoal−1.6/25.1−1.4/25.8−1.1/23.2−1.0/33.9−0.72/98.0
RacecarCircle14.1/15.00.7/14.112.3/15.911.9/13.826.57/30.4
DoggoCircle1.7/20.02.2/20.77.2/21.02.9/24.20.01/0.0

Entries report reward/cost interquartile means (IQM) across five seeds. The cost limit is 25. Bold: cost IQM ≤25.

W1 The vanilla baselines do not use observation stacking; we will make this explicit. We disagree that envstacking contributes much gains. Compared to TPRO-P, +envstacking trades the safety on one task for another whereas full SPEC meets the cost-IQM threshold on all tasks.

Seed variability is not unique to SPEC. It also affects the baselines. Deep RL is sensitive to initialization and stochasticity, while coupled nonlinear dynamics can produce oscillations.

W2 Our motivation is stated in the Introduction, which is to improve data efficiency by reusing historical cost experience while retaining conditional on-policy guarantees. Bias is a controlled consequence of this reuse, not the objective. The intuition of SPEC is to share sparse cost information, while clipping and adaptive weighting allow it to retain theoretical backing.

Ablations show critic L2, the five-cost-critic ensemble and random retrieval each exceed the budget on three tasks; none reproduces SPEC's six-task aggregate feasibility. These results establish SPEC's benefits beyond regularization, ensembling, or random memory reuse.

Refer to the rebuttal for Reviewer 5ND4 for sample efficiency numbers

W3 We respectfully disagree that the theory is merely a perturbation argument. SIF establishes admissible corrections and characterizes their effects on Bellman contraction, fixed-point error, trust-region guarantees, and primal–dual dynamics. Our contribution integrates these tools for controlled off-policy reuse in on-policy safe RL.

Loose worst-case bounds should not be conflated with weak empirical safety. SPEC achieves the most consistent aggregate constraint satisfaction among the compared methods. Tighter analytical bounds alone do not establish better downstream safety. For SPEC–TRPO-P, the relevant connection is the Lagrangian robustness analysis, complemented by empirical evaluation. See also our response to Reviewer KoU9.

W4 Base algorithm. We purposefully build on TRPO-P, the strongest method in our eight-algorithm evaluation, to test SPEC against a demanding baseline.

Cost-only correction. We disagree that cost-only correction is intrinsically a weakness. SPEC deliberately targets the cost critic because safety costs are often sparser than rewards (Section 4), while leaving reward targets unchanged. This is a design choice for SPEC, not a restriction of SIF.

Baseline coverage. We disagree that the baseline coverage is inadequate. SPEC is an on-policy safe RL method and is compared against eight SOTA approaches of the relevant comparison class. Though we also add off-policy experiments for the reviewer. All algorithms perform worse than the baseline TRPO-P. We report DDPG-P in the table above, and the others we ran (TD3-P, TD3-L and DDPG-L) performed even worse; omitted for space.

Reward collapse. We believe the reviewer has misunderstood the task-specific reward scales. Negative returns alone do not meant we don't avoid collapse. The relevant comparison is reward relative to the best-performing baseline on the same task.

Q. Computational overhead. We thank the reviewer for this suggestion. We report IQM wall-clock training hours for SPEC and TRPO-P.

TRPO-P → SPEC: DoggoButton 2.20 → 2.51, PointButton 1.51 → 2.22, PointGoal 1.95 → 2.09, CarGoal 2.04 → 2.16, RacecarCircle 1.65 → 1.99, DoggoCircle 1.64 → 1.97.

A full 2M-interaction run takes approximately 2–2.5 hours, with retrieval adding 7–43 minutes relative to TRPO-P. SPEC's retrieval memory is capped at 500k transitions, so its footprint plateaus at approximately 2.6–2.8 GB, depending on observation dimension. These bounded computational and memory costs keep SPEC practical while delivering sample-efficiency gains.

These ablations, theoretical guarantees, and efficiency results establish SPEC's benefits beyond generic critic corrections, with practical computational overhead.

I thank the authors for the additional experiments. Two concerns remain.

  1. Safety consistency and the role of retrieval are still not established. The rebuttal confirms that the vanilla baselines do not use observation stacking, so the fair comparison is against the new controls. Against these, SPEC’s advantage reduces to meeting the cost limit on 6/6 tasks versus 3/6. That count rests on margins well within seed variance, e.g. CarGoal 24.9 ± 16.9 versus 25.1 ± 7.5 for +envstack (β_max = 0, Table 2). No variance is reported for the other new controls.

Both the ensemble and random-retrieval controls obtain higher reward than SPEC on four of six tasks. On CarGoal, the ensemble beats SPEC on both reward and cost (−1.1/23.2 vs −1.5/24.9). This looks like a different point on the reward–cost trade-off rather than a gain specific to retrieval. The per-seed results (response to KoU9) also show only 2–4 of 5 SPEC seeds feasible on the Button and Goal tasks, which does not support “consistent constraint satisfaction”.

  1. The sample-efficiency claim remains unsupported. The joint-target metric is defined relative to the baseline’s final reward and is compared only against TRPO-P without stacking, so it inherits the same confound. By the authors’ own figures, SPEC reaches the cost limit later on four of six tasks. It also adds up to about 47% wall-clock time (PointButton: 1.51 → 2.22 h). The paper’s stated motivation, making on-policy safe RL more data-efficient, is therefore still not demonstrated.

I therefore maintain my rating.

We thank the reviewer for the continued engagement, but believe their assessment disproportionately emphasizes isolated weaknesses while overlooking SPEC's broader contributions and the additional evidence supporting our claims.

Fair comparisons and retrieval. Our requested ablations—including observation stacking, regularization, ensemble critics, and random retrieval—demonstrate that SPEC's improvements extend beyond architectural choices and warrant consideration.

Safety consistency and seed variance. We disagree that seed variance invalidates SPEC's safety improvements. SPEC satisfies the cost constraint on 6/6 tasks versus 3/6 for +envstack. Characterizing these differences as "marginal" overlooks a fundamental objective of safe RL: satisfying rather than violating safety constraints. To further address the reviewer's concerns about variance and per-seed feasibility, we report the number of seeds with cost ≤25 (+envstack vs. SPEC): DoggoButton 2/5 vs. 3/5, PointButton 2/5 vs. 4/5, PointGoal 2/5 vs. 3/5, CarGoal 2/5 vs. 2/5. Circle tasks are omitted due to space constraints and saturation.

The reviewer highlights that only 2–4 SPEC seeds are feasible, but overlooks that SPEC matches or outperforms TRPO-P (and now +envstack) on every task, achieving 12/20 feasible runs versus 8/20 for +envstack and 6/20 for TRPO-P. We never claimed perfect feasibility; our claim is improved relative safety consistency, despite the inherent variability of safe RL.

Additional variance reporting. We report reward/cost IQM ± SD across five seeds, bold indicating cost IQM ≤25. DDPG-P and +L2 .1 omitted due to poor and similar performance, respectively.

Env+envstackensTRPO-PRandom
DoggoButton0.21±0.10 / 27.8±7.50.15±0.12 / 28.4±7.30.17±0.07 / 21.1±4.7
PointButton−1.51±0.68 / 24.4±8.9−0.61±1.36 / 26.3±12.4−1.21±1.23 / 26.5±13.4
PointGoal−1.36±1.40 / 31.0±13.7−0.70±1.01 / 28.6±13.5−0.67±0.81 / 32.1±14.6
CarGoal−1.64±0.41 / 25.1±7.5−1.10±0.75 / 23.2±9.6−0.98±0.36 / 33.9±7.9
RacecarCircle14.08±3.64 / 15.0±10.512.26±8.12 / 15.9±22.411.86±5.92 / 13.8±6.7
DoggoCircle1.74±1.06 / 20.0±10.47.18±2.08 / 21.0±9.02.92±1.00 / 24.2±9.2

Ensemble and random reward comparison. The reviewer overlooks that the cited reward gains come with cost-limit violations, a fundamental concern in safe RL; CarGoal is the sole exception, but one task does not negate SPEC's broader cross-task consistency.

Sample efficiency and observation stacking. We disagree that vanilla TRPO-P is an invalid baseline: it measures the improvement of the complete SPEC algorithm over its underlying method. Nevertheless, we additionally compare against +envstack using the same reward target.

Joint-target time (M steps). Median steps to reach reward ≥R^ and cost ≤25 (+envstack vs. SPEC; successful seeds out of five in parentheses): DoggoButton 0.17 (5) vs. 0.12 (5), PointButton 1.24 (3) vs. 1.19 (4), PointGoal 0.77 (5) vs. 0.48 (5), CarGoal >2 (2) vs. 0.66 (4), RacecarCircle >2 (1) vs. 0.89 (3), DoggoCircle 1.78 (4) vs. 1.51 (4). R^ is TRPO-P's final reward IQM; >2 denotes censoring beyond 2M steps.

SPEC reaches the joint target earlier than +envstack on all six tasks, providing further evidence that its sample-efficiency improvements are not simply attributable to stacking.

Training time and computational efficiency. The reviewer highlights PointButton's 47% overhead while overlooking the broader results and our sample-efficiency evidence in response to 5ND4. We also show that SPEC is substantially faster than S-TRPO-P, the directly relevant safe episodic-control baseline, on all six tasks.

Training time. Wall-clock time for 2M steps, relative to TRPO-P (IQM over five seeds; lower is better; SPEC vs. S-TRPO-P): DoggoButton 1.14× vs. 1.89×, PointButton 1.47× vs. 1.89×, PointGoal 1.07× vs. 2.52×, CarGoal 1.06× vs. 2.46×, RacecarCircle 1.21× vs. 4.42×, DoggoCircle 1.20× vs. 4.45×.

SPEC incurs only 6–21% overhead on five tasks versus 89–345 for S-TRPO-P. Combined with the joint-target results, this demonstrates a favorable computational- and sample-efficiency trade-off.

Time to safety and reward collapse. The claim that SPEC reaches the cost threshold later on 4/6 tasks overlooks reward performance. S-TRPO-P reaches safety earlier but suffers severe reward collapse, which our paper explicitly addresses. The remaining cases involve TRPO-P, which ultimately violates the cost constraint on both tasks. Safety and reward must be evaluated jointly, rather than emphasizing time-to-safety in isolation.

Given the substantial additional evidence addressing the reviewer's concerns, we urge the reviewer to reconsider their interpretation of the paper and its conclusions in light of the full body of evidence. We remain happy to provide further comparisons during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.