- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how episodic information can augment on-policy safe reinforcement learning while controlling the resulting estimation bias. Its Safe Integration Framework (SIF) adds a clipped, Lipschitz correction to the lambda-return Bellman operator. The analysis establishes contraction and a fixed-point deviation bound, then propagates critic error into trust-region performance bounds and a local robustness result for primal–dual dynamics. The practical instantiation, Safe Proximity-guided Episodic Control (SPEC), retrieves historical trajectories in a learned cost-critic embedding, forms neighbor-based multi-step cost estimates, and uses their residuals to adjust critic targets. Corrections are weighted by IVF-cell concentration and annealed over training; the method also stacks environment observations. Experiments apply SPEC to TRPO-PID on six Safety-Gymnasium tasks with five seeds, comparing against safe-RL baselines and SafeEC, with ablations of correction strength, weighting, observation history, and encoder warmup.
Critical strengths
-
SIF makes the correction magnitude and sensitivity explicit rather than treating retrieved returns as reliable on-policy targets. Theorem 3.1 and the subsequent perturbation analysis provide a transparent way to reason about the cost of introducing auxiliary information. This is useful because stale or mismatched episodic information can otherwise introduce unaccounted-for bias into safe policy optimization.
-
SPEC is evaluated against both its base algorithm and a prior safe episodic-control method across six tasks. The beta-max-zero control retains environment observation stacking (Table 2), helping distinguish retrieval corrections from temporal-context changes. Table 1 shows potentially useful gains: PointButton cost falls from 32.3 to 18.1 relative to TRPO-PID at the same reported reward, and RacecarCircle reward is 14.8 versus SafeEC's 0.3 while SPEC's cost IQM remains below 25. These results support further investigation, even though they do not establish uniform superiority.
Critical weaknesses
-
The necessity and theoretical advance of the proposed approach over existing off-policy policy evaluation (OPE) methods are unclear. Appendix A mainly discusses safe RL and episodic control, without discussing directly relevant estimators such as Retrace and V-trace or explaining why adapting existing OPE methods to the cost critic would be insufficient. Such methods do not automatically solve constrained policy optimization, but they address the data-reuse problem motivating SPEC. SPEC instead adds a clipped historical-return residual to an on-policy target, accepting both policy-mismatch and state-neighborhood bias. This is not intrinsically invalid, but clipping does not establish more accurate estimates, a better bias–variance trade-off, or protection against cost underestimation. Theorems D.2 and 3.1 show that a sufficiently small Lipschitz perturbation preserves contraction to a generally biased fixed point; they do not establish that retrieval improves evaluation, and would also apply to uninformative corrections satisfying the same assumptions. The numerical guarantees appear weak: Table 6 parameters give worst-case value and GAE error bounds of approximately 110.7 and 3702.3 using beta_max and L_delta = 1, while Remark E.6 acknowledges potentially vacuous continuous-state performance bounds. Moreover, exact-operator convergence and conditional local ODE stability do not establish convergence of the implemented neural SPEC–TRPO-PID updates. Appendix E, Eq. (14), also applies the nominal performance inequality to a biased critic before accounting for baseline error, a step requiring justification. These issues leave the claimed benefit over existing evaluation methods and standard perturbation analysis insufficiently substantiated.
-
The empirical results do not convincingly establish improved safety consistency or a retrieval-specific advantage. Table 1 classifies methods as safe using cost IQM <= 25 across five seeds, yet SPEC's costs are 24.9 +/- 16.9 on CarGoal and 23.8 +/- 13.3 on RacecarCircle. Without per-seed feasibility statistics, cumulative training excess cost, and uncertainty estimates, these results provide limited evidence of reliable constraint satisfaction. The reward–cost trade-offs are also mixed: SPEC has lower reward and higher cost than TRPO-PID on DoggoCircle, and TRPO-Lag achieves higher reward and lower cost than SPEC on RacecarCircle. Furthermore, the beta-max-zero control already achieves RacecarCircle reward 14.1 and cost 15.0, versus 14.8 and 23.8 for full SPEC (Table 2). The absence of matched OPE, uniform-memory, or simple critic-regularization controls makes it difficult to attribute the gains to informative episodic retrieval rather than observation stacking or generic critic perturbation.
Questions for authors
-
What prevents existing OPE estimators from addressing the historical-data reuse component, and what does SPEC gain by accepting uncorrected bias? Please clarify whether the analysis establishes anything beyond stability under bounded perturbations, whether a quantitatively useful safety or estimation bound is available, and which guarantees apply to practical SPEC–TRPO-PID. Please also justify Eq. (14): in a one-state zero-reward MDP with identical policies and delta = 0, an admissible constant critic -b gives positive GAE and makes that intermediate inequality assert 0 >= b. Can the argument be repaired by starting from the nominal inequality and explicitly propagating the critic error?
-
Can you specify the evaluation window and aggregation procedure, and provide per-seed feasibility, cumulative training excess cost, and uncertainty estimates to substantiate improved safety consistency? Under matched architecture, observations, and training budget, how does SPEC compare with an appropriately adapted OPE estimator and a uniform-memory or simple critic-regularization control? These comparisons would help establish whether retrieval itself explains the reported gains.
3 comments
Thank you for the rebuttal. However, my main concern remains.
Off-policy policy-evaluation methods have established theoretical guarantees. Retrace and V-trace are two examples: their analyses establish contraction and characterize the value functions their estimators converge to under stated assumptions (Munos et al., 2016; Espeholt et al., 2018). Many other OPE methods also provide theoretical guarantees. Although these methods are often presented for reward critics, their evaluation framework can also be applied to cost critics: replacing the scalar reward with cost preserves the Bellman form, subject to the relevant assumptions.
Although these methods are commonly presented for reward critics, their evaluation operators have the same form for a cost critic: the scalar reward signal is replaced by the scalar cost signal. Thus, provided the relevant assumptions hold for the cost signal and the data, the corresponding policy-evaluation guarantees apply to cost estimation as well. If those assumptions cannot be met in the paper’s setting, the authors should explain which ones fail and why. The paper and the rebuttal have not provided this analysis. It does not sufficiently discuss existing OPE methods, identify assumptions that make them unsuitable for this setting, or explain why their limitations motivate SPEC. Nor does it establish what advantages SPEC offers over these alternatives in estimation accuracy or practical applicability.
We thank the reviewer for raising this point. Since the beginning of the discussion phase, we have been working on the theoretical and empirical analysis of Retrace and V-trace within TRPO-PID to clarify their relationship to SPEC. Both analyses are now complete, and we would be happy to share the results with the reviewers.
Adaptation to TRPO-PID. Replay trains only the cost critic; the reward critic retains on-policy training, the actor uses fresh rollouts, and PID uses observed episode costs. During evaluation, we freeze the reference policy and critic snapshot. Let V-trace uses , and
\delta_t^V&=c_t+g_tV_c^-(s_{t+1})-V_c^-(s_t),\ e_t^V&=\rho_t\delta_t^V+g_tz_t^Ve_{t+1}^V,\quad Y_t^V=V_c^-(s_t)+e_t^V. \end{aligned}$$ Retrace uses $z_t^Q=\lambda_{\mathrm{tr}}\min(1,w_t)$, $\bar V_c^-(s)=\mathbb E_{a\sim\pi_k}Q_c^-(s,a)$ and $$\begin{aligned} \delta_t^Q&=c_t+g_t\bar V_c^-(s_{t+1})-Q_c^-(s_t,a_t),\ e_t^Q&=\delta_t^Q+g_tz_{t+1}^Qe_{t+1}^Q,\quad Y_t^Q=Q_c^-(s_t,a_t)+e_t^Q. \end{aligned}$$ Recursions stop at segment boundaries; true terminations suppress bootstrapping, while time-limit truncations retain it. We fit the critic to detached targets. Fresh-rollout cost GAE uses $V_{c,\phi}$ for V-trace and $\mathbb E_{a\sim\pi_k}Q_{c,\phi}$ for Retrace, whose action expectations use Monte Carlo samples. The actor retains $\widehat A_\nu=(\widehat A_r-\nu_k\widehat A_c)/(1+\nu_k)$. **Theoretical analysis.** Appendix G of the revised manuscript contains the new analysis. We also incorporated Reviewer 5ND4's clarifications, applying Lemma E.2 directly to the nominal bound instead of the Eq. (14) step. We can share this revision. However, incorporating all additional experiments material will require more time. The trace evaluation operators are contractions under their original assumptions. Our extension connects their errors to SIF's bounds: given a valid nominal bound and uniform control of cost-critic error relative to the reference policy, the GAE perturbation and conditional trust-region performance arguments apply. Local Lagrangian robustness additionally requires the paper's idealized mean primal–dual ODE assumptions and sufficiently small critic-induced drift perturbations, uniformly in time. Retrace introduces no intrinsic clipped-target mismatch; its action expectation must be integrated, which adds integration error. V-trace requires accounting for its clipped-policy target or the actual mixed-replay fixed point. These guarantees extend beyond SPEC but remain conditional, without end-to-end guarantees for the implementations. We thank the reviewer for suggesting this analysis, which strengthens the paper. We now distinguish two components of SIF. First, its conditional policy-performance and local-robustness guarantees depend on a bound on cost-critic error relative to the reference policy and therefore extend to Retrace and V-trace under the required assumptions (Appendix G). Second, its fixed-point perturbation analysis addresses SPEC-type bounded additive corrections, a distinct mechanism from the Retrace and V-trace operators. This component provides an explicit worst-case deviation in terms of correction strength and cap. While this two-component distinction is not yet explicitly presented in the revised manuscript, we will incorporate it into the final version. Our evidence for SPEC’s practical advantage comes from the empirical comparison. **Empirical comparison.** Entries report reward IQM / cost IQM across five seeds. **Bold** indicates cost IQM ≤25. | Env | Retrace | V-trace | SPEC | |:---|---:|---:|---:| | DoggoButton | 0.27 / 33.6 | 0.30 / 36.0 | 0.17 / **24.1** | | PointButton | −1.45 / **20.1** | −1.19 / **22.1** | −1.29 / **18.1** | | PointGoal | −1.46 / 31.0 | −0.78 / 43.6 | −1.17 / **24.4** | | CarGoal | −0.54 / 27.5 | −0.94 / 32.3 | −1.49 / **24.9** | | RacecarCircle | 8.63 / 25.6 | 6.40 / **15.6** | 14.81 / **23.8** | | DoggoCircle | 2.37 / **21.7** | 2.10 / **22.1** | 2.48 / **18.3** | **SPEC achieves cost IQM ≤25 on all six tasks, versus two for TRPO-PID with a Retrace critic and three with a V-trace critic.** We did not directly evaluate critic accuracy because exact reference values are unavailable, and training MSE against noisy or bootstrapped targets alone does not establish true value-function accuracy. We will incorporate the full empirical comparison alongside the new theoretical extension in the revision. This will strengthen the paper through an additional theoretical contribution extending SIF’s conditional guarantees to Retrace and V-trace critics, establishing SIF’s broader applicability, and empirical comparisons reinforcing SPEC’s superior constraint-satisfaction profile.
Rebuttal for Reviewer KoU9
We thank the reviewer for their feedback but respectfully disagree with several interpretations of our contributions.
W1/Q1: Historical reuse and OPE. Retrace/V-trace can reuse historical costs under their assumptions, but do not themselves establish constrained policy guarantees. SPEC augments on-policy targets with bounded neighborhood estimates, sharing historical cost observations without behavior-policy ratios. Policy-mismatch and neighborhood bias remain, but that is the point of SIF, which asserts it's admissible. Clipping does not guarantee better accuracy, variance, or conservative costs, which is the point of SPEC. Our matched FQE comparison tests whether reuse alone explains SPEC's gains.
Contribution. Theorems D.2/3.1 establish contraction and bounded fixed-point deviation without requiring informative retrieval; they do not claim retrieval improves evaluation. This generality is intentional: SIF specifies conditions under which auxiliary information can be integrated while preserving conditional guarantees. Its scope also extends beyond critic stability: Theorem 3.2/Corollary E.4 bound reward/cost performance with explicit slack, while Appendix F analyzes local primal–dual dynamics. Retrieval's utility is evaluated empirically.
Bounds and practical scope. Our analysis deliberately prioritizes general admissibility and explicit control of perturbation effects. The submitted worst-case critic/GAE bounds are conservative: 110.7/3702.3 at β=0.15, remaining nonzero at β=0. Exploiting clipping and infinite-horizon GAE telescoping tightens these to 11.9/21.8 in the idealized setting, but doesn't certify safety. Table 2 provides evidence: βmax=0 meets the cost-IQM budget on three tasks, versus all six for SPEC. Thus, these guarantees concern idealized operators and conditional local dynamics, not convergence of neural actor–critic training, which is the purpose of SPEC.
Eq. (14). The counterexample highlights the need to account explicitly for baseline error. We will clarify the argument by starting from nominal Eq. (2) and applying Lemma E.2 to propagate critic error, preserving the conditional reward bound under its assumptions. We will also clarify cost-corollary GAE accumulation and associated slack; see our response to 5ND4 Q2.
W2/Q2: Safety consistency and evaluation. We report five seeds over 2M steps. Feasibility counts seeds with final cost ≤25. At 5k-step checkpoints, violation frequency is the fraction with cost >25; severity is mean excess C(t)−25 during violations. Both are seed IQMs. R/C SE denotes across-seed reward/cost standard error. Entries are TRPO-P → SPEC; near-saturated Circle tasks are omitted for space.
These results strengthen SPEC's safety case beyond aggregate IQM: final feasibility improves on three tasks and ties on one; violation frequency improves/ties on three, and severity improves on three. PointButton trades higher frequency for lower severity; CarGoal trades higher severity for fewer violations and greater feasibility. Reward SE falls on both Goal tasks, and cost SE falls except on CarGoal, where feasibility improves from 0/5 to 2/5.
Scope and task-level trade-offs. We disagree that mixed task-level outcomes negate SPEC's broader safety gains. Sections 4/6 acknowledge reward–cost trade-offs; uniform dominance is not claimed. The βmax=0 control retains SPEC's encoders and observation history but meets the budget on only three of six tasks. Its stronger RacecarCircle result does not outweigh SPEC's broader feasibility. The per-seed results above further support these safety benefits despite task-dependent trade-offs.
Attribution and matched controls. Our response to n5bM provides stacking, random-memory, and critic-regularization controls. These reinforce SPEC's contribution.
We additionally evaluate FQE–TRPO-P, fitting its cost critic via FQE on a uniform 500k-transition buffer, matching observation history, hidden layers, and 2M-step budget. Its reward/cost IQMs (five seeds) are DoggoButton 0.2/16.5, PointButton −1.3/31.0, PointGoal −0.4/28.6, CarGoal −1.6/25.4. SPEC meets the cost budget on all four, versus one for FQE–TRPO-P, with comparable/higher reward on three. This supports SPEC's integration mechanism beyond generic historical reuse, though isolating the causal role of theoretical guarantees requires further study.
SIF's conditional guarantees and SPEC's empirical gains provide an advance beyond standard perturbation analysis and generic off-policy reuse.