- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Multi-Horizon Safety Policy Coordination (MSPC), which learns and coordinates a family of safety-first policies specialized to different temporal horizons. It learns multiple safety-first policies using supervision over different future horizons, then coordinates their candidate actions online using shared reward and cost critics under a residual safety budget. The key insight is that safety preferences depend on the temporal scope of supervision and need not be nested across horizons. During deployment, shared reward and cost models evaluate candidate actions, and a coordinator selects actions according to the remaining budget. Action disagreement determines when coordination is refreshed.
Critical strengths
- This paper identifies useful temporal complementarity. Figure 2 and Section 6.1 show reversals between specialists.
- This paper introduces the notion of temporal safety specialization in offline safe RL and operationalizes it via horizon-dependent feasibility targets (finite-window max and a recursive global target).
- The disagreement-triggered refresh to gate coordination queries is simple and intuitive, with empirical evidence that higher disagreement correlates with larger critic score separation.
- The paper is easy to read, and the method is well-structured.
Critical weaknesses
- Limited incremental novelty beyond existing policy-switching methods. CAPS (arxiv: 2412.18946) already coordinates multiple policies using cost-based filtering and reward-based selection among feasible actions.
- MSPC acknowledges prior work that already leveraged a similar solution (policy-switching), so the main contribution in this paper is that it changes the source of policy diversity to horizon-dependent safety supervision, while adding disagreement-triggered refresh. However, comparisons against individual specialists, their safe-task union, and a smaller specialist set do not sufficiently distinguish temporal complementarity from the general benefit of having more candidate policies.
- Eq. (18) multiplies the raw remaining budget by the average discount weight. This corresponds to uniformly distributing cost over the remaining steps; it is not valid for arbitrary cost timing.
- The theoretical contribution is mostly elementary and provides limited support for the method’s central claims. Appendix B establishes many claims. Although the paper explicitly makes no formal safety guarantee, its theory contributes limited new insight into when the proposed coordination and refresh mechanisms should work.
- The residual-budget conversion to a discounted per-step allowance is heuristic, and its sensitivity/robustness are not analyzed; theoretical justification or alternatives are not thoroughly discussed.
- Missing comparisons to several recent baselines (e.g., B2R (2509.25727), CCAC (Constraint-Conditioned Actor-Critic for Offline Safe Reinforcement Learning), C2IQL (C2IQL: Constraint-Conditioned Implicit Q-learning for Safe Offline Reinforcement Learning))
- There are 38 tasks in the DSRL benchmark. But the paper only reports 18 tasks.
- Incomplete validation of key design choices and deployment-budget adaptability. Key ablations are missing, including an equally sized same-horizon ensemble and sensitivity to horizon sets or disagreement thresholds. All benchmark evaluations use , without testing the same trained controller at other deployment budgets, leaving its budget adaptability unverified.
Questions for authors
- How to derive the residual-budget allowance in Eq. (18)? Can you provide a formal justification and a sensitivity analysis to , episode length, and dataset mismatches (e.g., variable trajectory lengths)?
- How does Full4 compare with four independently trained specialists using the same supervision horizon, under matched candidate count, coordination, and training resources?
- To what extent do candidate actions remain in-distribution? Can you report per-specialist KL to the behavior policy and a measure of action support overlap to support the validity of shared-critic evaluation?
- Have you explored uncertainty-aware coordination (e.g., bootstrapped/ensemble Qc, quantile/upper-confidence costs) or conservative screening (e.g., add a margin to εt) to mitigate underestimated costs? How does it affect safety coverage and reward?
- For tasks with dense costs (Metadrive tasks in DSRL), how would you extend the finite-horizon targets beyond binary violation signals to capture frequency or magnitude without saturating labels at larger H?
12 comments
Reply to W4&W8: Theoretical Contributions and Design Sensitivity.
We thank the reviewer for raising questions about the theoretical support for MSPC and the sensitivity of its design choices. We clarify the role of our theoretical analysis and provide additional experiments examining both horizon-set selection and disagreement thresholds.
1. Theoretical contributions. Our theoretical analysis aims to characterize the structural mechanisms underlying MSPC. Proposition B.1 and Corollary B.2 establish how extending safety horizons introduces additional temporal information while potentially causing label saturation. Proposition B.3 shows how horizon-dependent feasibility advantages can induce distinct policies from the same offline behavior distribution, providing a theoretical basis for temporal safety specialization. Proposition B.4 further establishes that small action disagreement bounds critic-score separation under a Lipschitz assumption, motivating disagreement-based query efficiency. Together, these results explain the mechanisms behind MSPC's design, while our empirical findings demonstrate non-nested closed-loop safety and the benefits of coordinating complementary temporal specialists.
2. Horizon-set sensitivity. The default H32 is selected using the dataset-informed label-novelty criterion in Appendix D.2 (Table 6), rather than downstream performance tuning. Specifically, H32 achieves higher label novelty (0.293) than H16 (0.145) and H64 (0.167), and is selected in all 500 trajectory-level bootstrap resamples. We further evaluate alternative horizon sets by replacing the intermediate H32 with H16 or H64, while retaining H8, H128, and Global and keeping the number of specialists and coordination settings unchanged. As shown in the following table, all three configurations maintain broad safety coverage, satisfying 15/18, 15/18, and 13/18 tasks with H16, H32, and H64, respectively. The different horizon sets exhibit distinct reward–cost trade-offs: H16 achieves higher rewards with the same overall safe-task coverage as H32 (15/18), while H32 yields lower mean costs on both suites. These results indicate that MSPC's effectiveness does not critically depend on a particular intermediate horizon, although its reward-safety operating point varies with the chosen temporal supervision.
Table: Sensitivity to Intermediate Safety Horizons. H8, H128, and Global (G) are fixed, while the intermediate horizon varies among H16, H32, and H64.
| Horizon Set | Bullet Reward ↑ | Bullet Cost ↓ | Bullet Safe | Safety-Gym. Reward ↑ | Safety-Gym. Cost ↓ | Safety-Gym. Safe | Total Safe ↑ |
|---|---|---|---|---|---|---|---|
| {8, 16, 128, G} | 0.460 ± 0.043 | 0.696 ± 0.175 | 6/8 | 0.299 ± 0.035 | 0.596 ± 0.364 | 9/10 | 15/18 |
| {8, 32, 128, G} (Ours) | 0.389 ± 0.072 | 0.308 ± 0.252 | 7/8 | 0.291 ± 0.028 | 0.498 ± 0.227 | 8/10 | 15/18 |
| {8, 64, 128, G} | 0.395 ± 0.070 | 0.439 ± 0.367 | 6/8 | 0.292 ± 0.029 | 0.789 ± 0.574 | 7/10 | 13/18 |
3. Disagreement-threshold sensitivity. We additionally evaluate disagreement thresholds calibrated at the 10th, 25th, 50th, and 75th percentiles of the offline action-disagreement distribution, alongside every-step coordination. The following table reports the resulting reward, cost, and coordination-query rates.
Table: Sensitivity to Disagreement Thresholds. Reward and normalized cost are reported as suite-level means ± average task-wise standard deviations across actor–critic configurations. Query rate denotes the percentage of environment steps requiring critic-based coordination.
| Threshold | Bullet Reward ↑ | Bullet Cost ↓ | Bullet Query (%) ↓ | Safety-Gym. Reward ↑ | Safety-Gym. Cost ↓ | Safety-Gym. Query (%) ↓ |
|---|---|---|---|---|---|---|
| Every-step | 0.400 ± 0.073 | 0.354 ± 0.243 | 100.0 | 0.290 ± 0.026 | 0.430 ± 0.194 | 100.0 |
| p10 | 0.410 ± 0.065 | 0.411 ± 0.267 | 92.8 | 0.290 ± 0.029 | 0.477 ± 0.297 | 86.6 |
| p25 (Ours) | 0.389 ± 0.072 | 0.308 ± 0.252 | 81.8 | 0.291 ± 0.028 | 0.498 ± 0.227 | 70.5 |
| p50 | 0.393 ± 0.061 | 0.599 ± 0.374 | 62.7 | 0.294 ± 0.026 | 0.585 ± 0.292 | 44.9 |
| p75 | 0.400 ± 0.061 | 0.646 ± 0.313 | 40.0 | 0.300 ± 0.031 | 0.631 ± 0.400 | 21.9 |
The results reveal a safety–efficiency trade-off as the disagreement threshold varies. Lower thresholds generally increase coordination frequency, whereas higher thresholds reduce critic queries but may increase safety costs. On Bullet, the default p25 achieves the lowest mean cost (0.308) while reducing coordination queries by 18.2% relative to every-step selection. On Safety-Gymnasium, p25 reduces the query rate from 86.6% to 70.5% compared with p10, with a modest increase in mean cost (0.477 to 0.498) and comparable reward. Further increasing the threshold to p50 or p75 reduces coordination overhead more substantially but leads to higher mean costs on both suites. Overall, p25 offers a practical safety–efficiency trade-off, and MSPC remains effective across multiple horizon sets.
Reply to W3&W5&Q1: Residual-Budget Derivation and Sensitivity. (Part 1)
We clarify the derivation of Eq. (18) and systematically examine how cost timing, discounting, episode-duration estimation, and dataset characteristics affect the resulting coordination behavior. Our additional analyses combine an explicit mathematical interpretation with controlled evaluations across all 18 tasks.
1. Derivation and interpretation. Eq. (18) follows the average-discount-factor conversion adopted in prior safe RL methods, including FAC, CVPO, and CAPS. Given a remaining raw-cost budget and estimated remaining horizon , we first allocate the budget uniformly across the remaining steps and express this per-step allowance in the discounted units used by the cost critic:
Define , with . Under a fixed remaining horizon and uniform conditional expected per-step costs, , the discounted and raw expected continuation costs satisfy
This establishes the exact correspondence under the stated condition. For nonuniform costs, the conversion instead induces an explicit temporal weighting : near-term risks receive greater weight, while more distant risks can be reassessed as new observations arrive and the residual budget is updated. Thus, Eq. (18) provides a discount-consistent screening heuristic whose temporal emphasis is directly controlled by .
2. Sensitivity to cost discounting. We evaluate across all 18 tasks at . For each setting, we train a cost critic with the corresponding discount factor and apply its discount-consistent residual-budget allowance. The undiscounted setting uses . All temporal specialists and reward critics remain frozen.
As shown in the following table, cost discounting provides a meaningful control over the reward--safety operating point. On Bullet, increasing progressively reduces mean normalized cost from 0.803 to 0.157, accompanied by a reward decrease from 0.490 to 0.314. On Safety-Gymnasium, the default achieves the lowest mean cost (0.498) among all tested settings, while maintaining higher reward than the less-discounted alternatives and . In particular, the undiscounted variant achieves both lower reward (0.255 vs. 0.291) and higher mean cost (0.607 vs. 0.498) on Safety-Gymnasium. These results demonstrate that stronger weighting of distant risks does not necessarily improve closed-loop safety, consistent with our broader finding that temporal safety preferences induce distinct, task-dependent behaviors.
Table: Cost-Discount Sensitivity at B = 10. Reward and normalized cost are reported as suite-level means ± average task-wise standard deviations across actor–critic configurations.
| γc | Bullet Reward ↑ | Bullet Cost ↓ | Safety-Gym. Reward ↑ | Safety-Gym. Cost ↓ |
|---|---|---|---|---|
| 0.97 | 0.490 ± 0.044 | 0.803 ± 0.227 | 0.312 ± 0.039 | 0.748 ± 0.393 |
| 0.99 (Ours) | 0.389 ± 0.072 | 0.308 ± 0.252 | 0.291 ± 0.028 | 0.498 ± 0.227 |
| 0.995 | 0.368 ± 0.064 | 0.168 ± 0.090 | 0.265 ± 0.030 | 0.544 ± 0.279 |
| 1.00 | 0.314 ± 0.064 | 0.157 ± 0.062 | 0.255 ± 0.028 | 0.607 ± 0.502 |
Reply to W1: Novelty and Distinction from Existing Methods, and W2&Q2&W8: Same-Horizon Ensemble Comparison.
1. W1: Novelty and Distinction from Existing Methods. We thank the reviewer for highlighting the relationship between MSPC and CAPS. While CAPS focuses on adapting to varying safety budgets by switching among policies with different reward-cost preferences, MSPC investigates a distinct and fundamental question in offline safe RL: Does extending the safety-supervision horizon necessarily lead to safer closed-loop behavior, or can different horizons offer complementary safety advantages?
Our key finding is that nested safety supervision does not imply monotonic closed-loop safety, and different horizons induce complementary safety behaviors (Fig.2). To our knowledge, this temporal safety complementarity and its exploitation have not been systematically investigated in prior offline safe RL work. Motivated by this finding, MSPC constructs horizon-specialized, safety-first policies and coordinates their actions under the remaining safety budget. While building on the policy-switching mechanism similar to CAPS, MSPC introduces a distinct source of policy diversity: temporal safety preferences induced by distinct supervision horizons, rather than different reward--cost preferences.
Furthermore, CAPS reports its main results averaged over multiple budgets, including 20,40,80 for Safety-Gymnasium. Under a common, stringent evaluation budget of B=10, MSPC satisfies the mean-cost constraint on 8/10 tasks compared with 6/10 for CAPS-IQL, while achieving substantially lower mean normalized cost (0.50 vs. 1.54). These results further support the effectiveness of our safety-first, multi-horizon design.
2. W2&Q2&W8: Same-Horizon Ensemble Comparison.
We thank the reviewer for pointing out this question, and we agree that our original comparisons with individual specialists and H8+Global do not fully disentangle the benefits of temporal specialization from those of increasing the number of candidate policies. To directly address this concern, we introduce four matched same-horizon ensembles (H8$\times$4, H32$\times$4, H128$\times$4, and Global$\times$4), each comprising four independently trained specialists with identical horizon supervision but different random initializations. All comparisons match the candidate count, model capacity, training resources, and coordination mechanism, isolating the contribution of horizon-dependent supervision from generic ensemble diversity.
As shown in the following table, MSPC achieves the lowest mean cost on both benchmark suites (0.308 on Bullet and 0.498 on Safety-Gymnasium) and the highest overall safety coverage (15/18 tasks), compared with at most 14/18 for same-horizon ensembles. On Bullet, Full4 reduces mean cost to 0.308, compared with at least 0.765 for same-horizon ensembles, while improving safety coverage from 6/8 to 7/8 tasks. On Safety-Gymnasium, Full4 also achieves the lowest mean cost (0.498 vs. at least 0.595) and jointly highest safety coverage (8/10).
Table: Mixed-Horizon MSPC versus Same-Horizon Ensembles. Comparison of mixed-horizon MSPC with four-policy same-horizon ensembles under matched model capacity and training resources.
Notably, Global$\times$4 is the only same-horizon ensemble achieving suite-average normalized costs below 1 on both benchmarks. Nevertheless, Full4 achieves substantially lower costs on both suites (0.308/0.498 vs. 0.821/0.595), slightly higher overall task-averaged reward (0.335 vs. 0.329), and broader safety coverage (15/18 vs. 13/18).
These controlled results provide direct empirical evidence that horizon-dependent safety supervision produces complementary behaviors beyond those induced by random initialization alone. The improvement therefore cannot be attributed merely to increasing the number of candidate policies or model capacity, strengthening our central claim that structured temporal safety diversity contributes to more effective policy coordination.