acceptodds
Under review as a conference paper at ICLR 2027

Beyond a Single Safety Horizon: Multi-Horizon Policy Coordination for Offline Safe Reinforcement Learning

Abstract

Safe offline reinforcement learning aims to maximize task performance under safety constraints using only static datasets. Existing approaches typically deploy a single policy, collapsing potentially different temporal safety preferences into one behavior. However, safety violations can unfold over substantially different time scales, such that changing the safety horizon alters which future hazards are visible when evaluating an action and can induce distinct safety-oriented behaviors. This motivates us to exploit temporal safety specialization rather than compressing multiple safety scales into a single policy. We propose **Multi-Horizon Safety Policy Coordination (MSPC)**, which learns and coordinates a family of safety-first policies specialized to different temporal horizons. At the low level, safety supervision over multiple future horizons is used to extract complementary, behavior-supported policies from the same offline dataset. At the high level, shared reward and cost critics evaluate their candidate actions, and a residual-budget coordinator selects among them based on the remaining safety allowance. We further use policy disagreement to avoid redundant coordination when candidate actions remain similar. Experiments on 18 tasks from two offline safe RL benchmark suites show that the evaluated fixed specialists exhibit non-nested safety outcomes, highlighting the need for adaptive coordination among complementary temporal safety behaviors. MSPC further achieves favorable reward–safety performance against representative offline safe RL baselines, demonstrating the effectiveness of temporal safety specialization and budget-aware action-level coordination.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Offline Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_X2mgOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Multi-Horizon Safety Policy Coordination (MSPC), which learns and coordinates a family of safety-first policies specialized to different temporal horizons. It learns multiple safety-first policies using supervision over different future horizons, then coordinates their candidate actions online using shared reward and cost critics under a residual safety budget. The key insight is that safety preferences depend on the temporal scope of supervision and need not be nested across horizons. During deployment, shared reward and cost models evaluate candidate actions, and a coordinator selects actions according to the remaining budget. Action disagreement determines when coordination is refreshed.

Critical strengths
  1. This paper identifies useful temporal complementarity. Figure 2 and Section 6.1 show reversals between specialists.
  2. This paper introduces the notion of temporal safety specialization in offline safe RL and operationalizes it via horizon-dependent feasibility targets (finite-window max and a recursive global target).
  3. The disagreement-triggered refresh to gate coordination queries is simple and intuitive, with empirical evidence that higher disagreement correlates with larger critic score separation.
  4. The paper is easy to read, and the method is well-structured.
Critical weaknesses
  1. Limited incremental novelty beyond existing policy-switching methods. CAPS (arxiv: 2412.18946) already coordinates multiple policies using cost-based filtering and reward-based selection among feasible actions.
  2. MSPC acknowledges prior work that already leveraged a similar solution (policy-switching), so the main contribution in this paper is that it changes the source of policy diversity to horizon-dependent safety supervision, while adding disagreement-triggered refresh. However, comparisons against individual specialists, their safe-task union, and a smaller specialist set do not sufficiently distinguish temporal complementarity from the general benefit of having more candidate policies.
  3. Eq. (18) multiplies the raw remaining budget by the average discount weight. This corresponds to uniformly distributing cost over the remaining steps; it is not valid for arbitrary cost timing.
  4. The theoretical contribution is mostly elementary and provides limited support for the method’s central claims. Appendix B establishes many claims. Although the paper explicitly makes no formal safety guarantee, its theory contributes limited new insight into when the proposed coordination and refresh mechanisms should work.
  5. The residual-budget conversion to a discounted per-step allowance is heuristic, and its sensitivity/robustness are not analyzed; theoretical justification or alternatives are not thoroughly discussed.
  6. Missing comparisons to several recent baselines (e.g., B2R (2509.25727), CCAC (Constraint-Conditioned Actor-Critic for Offline Safe Reinforcement Learning), C2IQL (C2IQL: Constraint-Conditioned Implicit Q-learning for Safe Offline Reinforcement Learning))
  7. There are 38 tasks in the DSRL benchmark. But the paper only reports 18 tasks.
  8. Incomplete validation of key design choices and deployment-budget adaptability. Key ablations are missing, including an equally sized same-horizon ensemble and sensitivity to horizon sets or disagreement thresholds. All benchmark evaluations use B=10B=10, without testing the same trained controller at other deployment budgets, leaving its budget adaptability unverified.
Questions for authors
  1. How to derive the residual-budget allowance ϵt\epsilon_t in Eq. (18)? Can you provide a formal justification and a sensitivity analysis to γ\gamma, episode length, and dataset mismatches (e.g., variable trajectory lengths)?
  2. How does Full4 compare with four independently trained specialists using the same supervision horizon, under matched candidate count, coordination, and training resources?
  3. To what extent do candidate actions remain in-distribution? Can you report per-specialist KL to the behavior policy and a measure of action support overlap to support the validity of shared-critic evaluation?
  4. Have you explored uncertainty-aware coordination (e.g., bootstrapped/ensemble Qc, quantile/upper-confidence costs) or conservative screening (e.g., add a margin to εt) to mitigate underestimated costs? How does it affect safety coverage and reward?
  5. For tasks with dense costs (Metadrive tasks in DSRL), how would you extend the finite-horizon targets beyond binary violation signals to capture frequency or magnitude without saturating labels at larger H?
12 comments
AuthorsOpenReview ↗

Reply to W1: Novelty and Distinction from Existing Methods, and W2&Q2&W8: Same-Horizon Ensemble Comparison.

1. W1: Novelty and Distinction from Existing Methods. We thank the reviewer for highlighting the relationship between MSPC and CAPS. While CAPS focuses on adapting to varying safety budgets by switching among policies with different reward-cost preferences, MSPC investigates a distinct and fundamental question in offline safe RL: Does extending the safety-supervision horizon necessarily lead to safer closed-loop behavior, or can different horizons offer complementary safety advantages?

Our key finding is that nested safety supervision does not imply monotonic closed-loop safety, and different horizons induce complementary safety behaviors (Fig.2). To our knowledge, this temporal safety complementarity and its exploitation have not been systematically investigated in prior offline safe RL work. Motivated by this finding, MSPC constructs horizon-specialized, safety-first policies and coordinates their actions under the remaining safety budget. While building on the policy-switching mechanism similar to CAPS, MSPC introduces a distinct source of policy diversity: temporal safety preferences induced by distinct supervision horizons, rather than different reward--cost preferences.

Furthermore, CAPS reports its main results averaged over multiple budgets, including 20,40,80\\{20,40,80\\} for Safety-Gymnasium. Under a common, stringent evaluation budget of B=10B=10, MSPC satisfies the mean-cost constraint on 8/10 tasks compared with 6/10 for CAPS-IQL, while achieving substantially lower mean normalized cost (0.50 vs. 1.54). These results further support the effectiveness of our safety-first, multi-horizon design.

2. W2&Q2&W8: Same-Horizon Ensemble Comparison.

We thank the reviewer for pointing out this question, and we agree that our original comparisons with individual specialists and H8+Global do not fully disentangle the benefits of temporal specialization from those of increasing the number of candidate policies. To directly address this concern, we introduce four matched same-horizon ensembles (H8$\times$4, H32$\times$4, H128$\times$4, and Global$\times$4), each comprising four independently trained specialists with identical horizon supervision but different random initializations. All comparisons match the candidate count, model capacity, training resources, and coordination mechanism, isolating the contribution of horizon-dependent supervision from generic ensemble diversity.

As shown in the following table, MSPC achieves the lowest mean cost on both benchmark suites (0.308 on Bullet and 0.498 on Safety-Gymnasium) and the highest overall safety coverage (15/18 tasks), compared with at most 14/18 for same-horizon ensembles. On Bullet, Full4 reduces mean cost to 0.308, compared with at least 0.765 for same-horizon ensembles, while improving safety coverage from 6/8 to 7/8 tasks. On Safety-Gymnasium, Full4 also achieves the lowest mean cost (0.498 vs. at least 0.595) and jointly highest safety coverage (8/10).

Table: Mixed-Horizon MSPC versus Same-Horizon Ensembles. Comparison of mixed-horizon MSPC with four-policy same-horizon ensembles under matched model capacity and training resources.

Policy FamilyBullet Reward ↑Bullet Cost ↓Bullet SafeSafety-Gym. Reward ↑Safety-Gym. Cost ↓Safety-Gym. SafeTotal Safe ↑
Same-horizon ensembles
H8 × 40.537 ± 0.0211.201 ± 0.1326/80.333 ± 0.0250.863 ± 0.3148/1014/18
H32 × 40.415 ± 0.0440.806 ± 0.2396/80.295 ± 0.0251.055 ± 0.2976/1012/18
H128 × 40.332 ± 0.0770.765 ± 0.2676/80.308 ± 0.0211.345 ± 0.6536/1012/18
Global × 40.417 ± 0.0210.821 ± 0.2636/80.258 ± 0.0190.595 ± 0.3387/1013/18
Mixed-horizon specialization
MSPC (Full4)0.389 ± 0.0720.308 ± 0.2527/80.291 ± 0.0280.498 ± 0.2278/1015/18

Notably, Global$\times$4 is the only same-horizon ensemble achieving suite-average normalized costs below 1 on both benchmarks. Nevertheless, Full4 achieves substantially lower costs on both suites (0.308/0.498 vs. 0.821/0.595), slightly higher overall task-averaged reward (0.335 vs. 0.329), and broader safety coverage (15/18 vs. 13/18).

These controlled results provide direct empirical evidence that horizon-dependent safety supervision produces complementary behaviors beyond those induced by random initialization alone. The improvement therefore cannot be attributed merely to increasing the number of candidate policies or model capacity, strengthening our central claim that structured temporal safety diversity contributes to more effective policy coordination.

AuthorsOpenReview ↗

Reply to W4&W8: Theoretical Contributions and Design Sensitivity.

We thank the reviewer for raising questions about the theoretical support for MSPC and the sensitivity of its design choices. We clarify the role of our theoretical analysis and provide additional experiments examining both horizon-set selection and disagreement thresholds.

1. Theoretical contributions. Our theoretical analysis aims to characterize the structural mechanisms underlying MSPC. Proposition B.1 and Corollary B.2 establish how extending safety horizons introduces additional temporal information while potentially causing label saturation. Proposition B.3 shows how horizon-dependent feasibility advantages can induce distinct policies from the same offline behavior distribution, providing a theoretical basis for temporal safety specialization. Proposition B.4 further establishes that small action disagreement bounds critic-score separation under a Lipschitz assumption, motivating disagreement-based query efficiency. Together, these results explain the mechanisms behind MSPC's design, while our empirical findings demonstrate non-nested closed-loop safety and the benefits of coordinating complementary temporal specialists.

2. Horizon-set sensitivity. The default H32 is selected using the dataset-informed label-novelty criterion in Appendix D.2 (Table 6), rather than downstream performance tuning. Specifically, H32 achieves higher label novelty (0.293) than H16 (0.145) and H64 (0.167), and is selected in all 500 trajectory-level bootstrap resamples. We further evaluate alternative horizon sets by replacing the intermediate H32 with H16 or H64, while retaining H8, H128, and Global and keeping the number of specialists and coordination settings unchanged. As shown in the following table, all three configurations maintain broad safety coverage, satisfying 15/18, 15/18, and 13/18 tasks with H16, H32, and H64, respectively. The different horizon sets exhibit distinct reward–cost trade-offs: H16 achieves higher rewards with the same overall safe-task coverage as H32 (15/18), while H32 yields lower mean costs on both suites. These results indicate that MSPC's effectiveness does not critically depend on a particular intermediate horizon, although its reward-safety operating point varies with the chosen temporal supervision.

Table: Sensitivity to Intermediate Safety Horizons. H8, H128, and Global (G) are fixed, while the intermediate horizon varies among H16, H32, and H64.

Horizon SetBullet Reward ↑Bullet Cost ↓Bullet SafeSafety-Gym. Reward ↑Safety-Gym. Cost ↓Safety-Gym. SafeTotal Safe ↑
{8, 16, 128, G}0.460 ± 0.0430.696 ± 0.1756/80.299 ± 0.0350.596 ± 0.3649/1015/18
{8, 32, 128, G} (Ours)0.389 ± 0.0720.308 ± 0.2527/80.291 ± 0.0280.498 ± 0.2278/1015/18
{8, 64, 128, G}0.395 ± 0.0700.439 ± 0.3676/80.292 ± 0.0290.789 ± 0.5747/1013/18

3. Disagreement-threshold sensitivity. We additionally evaluate disagreement thresholds calibrated at the 10th, 25th, 50th, and 75th percentiles of the offline action-disagreement distribution, alongside every-step coordination. The following table reports the resulting reward, cost, and coordination-query rates.

Table: Sensitivity to Disagreement Thresholds. Reward and normalized cost are reported as suite-level means ± average task-wise standard deviations across actor–critic configurations. Query rate denotes the percentage of environment steps requiring critic-based coordination.

ThresholdBullet Reward ↑Bullet Cost ↓Bullet Query (%) ↓Safety-Gym. Reward ↑Safety-Gym. Cost ↓Safety-Gym. Query (%) ↓
Every-step0.400 ± 0.0730.354 ± 0.243100.00.290 ± 0.0260.430 ± 0.194100.0
p100.410 ± 0.0650.411 ± 0.26792.80.290 ± 0.0290.477 ± 0.29786.6
p25 (Ours)0.389 ± 0.0720.308 ± 0.25281.80.291 ± 0.0280.498 ± 0.22770.5
p500.393 ± 0.0610.599 ± 0.37462.70.294 ± 0.0260.585 ± 0.29244.9
p750.400 ± 0.0610.646 ± 0.31340.00.300 ± 0.0310.631 ± 0.40021.9

The results reveal a safety–efficiency trade-off as the disagreement threshold varies. Lower thresholds generally increase coordination frequency, whereas higher thresholds reduce critic queries but may increase safety costs. On Bullet, the default p25 achieves the lowest mean cost (0.308) while reducing coordination queries by 18.2% relative to every-step selection. On Safety-Gymnasium, p25 reduces the query rate from 86.6% to 70.5% compared with p10, with a modest increase in mean cost (0.477 to 0.498) and comparable reward. Further increasing the threshold to p50 or p75 reduces coordination overhead more substantially but leads to higher mean costs on both suites. Overall, p25 offers a practical safety–efficiency trade-off, and MSPC remains effective across multiple horizon sets.

AuthorsOpenReview ↗

Reply to W3&W5&Q1: Residual-Budget Derivation and Sensitivity. (Part 1)

We clarify the derivation of Eq. (18) and systematically examine how cost timing, discounting, episode-duration estimation, and dataset characteristics affect the resulting coordination behavior. Our additional analyses combine an explicit mathematical interpretation with controlled evaluations across all 18 tasks.

1. Derivation and interpretation. Eq. (18) follows the average-discount-factor conversion adopted in prior safe RL methods, including FAC, CVPO, and CAPS. Given a remaining raw-cost budget btb _ {t} and estimated remaining horizon ntn _ {t}, we first allocate the budget uniformly across the remaining steps and express this per-step allowance in the discounted units used by the cost critic:

ϵt=∑j=0nt−1γcjbtnt=bt1−γcnt(1−γc)nt.\epsilon _ {t}=\sum _ {j=0} ^ {n _ {t}-1}\gamma _ {c} ^ {j}\frac{b _ {t}}{n _ {t}}=b _ {t}\frac{1-\gamma _ {c} ^ {n _ {t}}}{(1-\gamma _ {c})n _ {t}}.

Define κγc(n)=1n∑j=0n−1γcj\kappa _ {\gamma _ {c}}(n)=\frac{1}{n}\sum _ {j=0} ^ {n-1}\gamma _ {c} ^ {j}, with κ1(n)=1\kappa _ {1}(n)=1. Under a fixed remaining horizon and uniform conditional expected per-step costs, E[ct+j∣It]=μt\mathbb{E}[c _ {t+j}\mid\mathcal{I} _ {t}]=\mu _ {t}, the discounted and raw expected continuation costs satisfy

E[Gtγc∣It]=κγc(nt)E[Gtraw∣It].\mathbb{E}[G _ {t} ^ {\gamma _ {c}}\mid\mathcal{I} _ {t}]=\kappa _ {\gamma _ {c}}(n _ {t})\mathbb{E}[G _ {t} ^ {\mathrm{raw}}\mid\mathcal{I} _ {t}].

This establishes the exact correspondence under the stated condition. For nonuniform costs, the conversion instead induces an explicit temporal weighting wj=γcj/κγc(nt)w _ {j}=\gamma _ {c} ^ {j}/\kappa _ {\gamma _ {c}}(n _ {t}): near-term risks receive greater weight, while more distant risks can be reassessed as new observations arrive and the residual budget is updated. Thus, Eq. (18) provides a discount-consistent screening heuristic whose temporal emphasis is directly controlled by γc\gamma _ {c}.

2. Sensitivity to cost discounting. We evaluate γc∈0.97,0.99,0.995,1\gamma_c\in\\{0.97,0.99,0.995,1\\} across all 18 tasks at B=10B=10. For each setting, we train a cost critic with the corresponding discount factor and apply its discount-consistent residual-budget allowance. The undiscounted setting uses ϵt=bt\epsilon_t=b_t. All temporal specialists and reward critics remain frozen.

As shown in the following table, cost discounting provides a meaningful control over the reward--safety operating point. On Bullet, increasing γc\gamma_c progressively reduces mean normalized cost from 0.803 to 0.157, accompanied by a reward decrease from 0.490 to 0.314. On Safety-Gymnasium, the default γc=0.99\gamma_c=0.99 achieves the lowest mean cost (0.498) among all tested settings, while maintaining higher reward than the less-discounted alternatives γc=0.995\gamma_c=0.995 and 11. In particular, the undiscounted variant achieves both lower reward (0.255 vs. 0.291) and higher mean cost (0.607 vs. 0.498) on Safety-Gymnasium. These results demonstrate that stronger weighting of distant risks does not necessarily improve closed-loop safety, consistent with our broader finding that temporal safety preferences induce distinct, task-dependent behaviors.

Table: Cost-Discount Sensitivity at B = 10. Reward and normalized cost are reported as suite-level means ± average task-wise standard deviations across actor–critic configurations.

γcBullet Reward ↑Bullet Cost ↓Safety-Gym. Reward ↑Safety-Gym. Cost ↓
0.970.490 ± 0.0440.803 ± 0.2270.312 ± 0.0390.748 ± 0.393
0.99 (Ours)0.389 ± 0.0720.308 ± 0.2520.291 ± 0.0280.498 ± 0.227
0.9950.368 ± 0.0640.168 ± 0.0900.265 ± 0.0300.544 ± 0.279
1.000.314 ± 0.0640.157 ± 0.0620.255 ± 0.0280.607 ± 0.502
Reviewer_CP96OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Multi-Horizon Safety Policy Coordination (MSPC) for offline safe reinforcement learning. The main motivation is that safety supervision over different temporal horizons may induce distinct policies, and a longer safety horizon does not necessarily lead to safer closed-loop behavior. To exploit this temporal complementarity, MSPC learns a set of safety-oriented policies using finite-horizon and recursively propagated global feasibility targets. At deployment, a high-level coordinator uses shared reward and cost critics to select among the candidate actions based on the remaining safety budget. The method also introduces a policy-disagreement mechanism to reduce unnecessary coordination queries. The authors evaluate MSPC on 18 tasks from Bullet-Safety-Gym and Safety-Gymnasium, comparing it against seven offline safe RL baselines. MSPC satisfies the safety constraints on 15 of the 18 tasks while maintaining competitive returns. The experiments further show non-nested safety performance across individual horizon-specialized policies, with MSPC achieving broader safe-task coverage than the union of the fixed specialists (15 versus 12 tasks). Ablation studies examine the contribution of intermediate safety horizons and disagreement-triggered coordination, providing additional evidence for the effectiveness of the proposed framework.

Critical strengths
  1. Well-motivated multi-horizon safety specialization and coordination. The paper presents a coherent approach to addressing the limitations of relying on a single safety horizon in offline safe RL. By learning safety-oriented policies from different temporal feasibility targets and coordinating their actions using shared reward and cost critics, MSPC can exploit complementary safety behaviors without assuming that longer horizons necessarily produce safer policies. The separation between temporal safety supervision and deployment-time action selection is a reasonable design choice, and the accompanying theoretical analysis provides additional justification for the framework.

  2. Extensive empirical evaluation with meaningful ablation studies. The authors evaluate MSPC on 18 tasks across two benchmark suites against seven offline safe RL baselines, demonstrating competitive returns and improved safety constraint satisfaction. Importantly, the experiments go beyond aggregate performance comparisons. The analysis of individual horizon-specialized policies demonstrates their complementary safety behaviors, while the comparison between H8+Global and Full4 provides direct evidence for the contribution of intermediate safety horizons. The coordination analysis further examines the effectiveness of disagreement-triggered refresh. These results provide reasonably strong empirical support for the main design choices and are an important positive factor in my assessment.

Critical weaknesses
  1. Reliability of residual-budget coordination. The coordinator compares an offline-learned discounted cost-to-go estimate against an allowance converted from the remaining undiscounted episodic budget (Eqs. 18–20). The conversion in Eq. (18) scales the remaining budget by the average discount factor over the remaining episode, but this does not generally preserve the episodic cost constraint when costs are unevenly distributed over time. Moreover, the shared IQL-style cost critic is trained without accounting for the future action-selection decisions of the deployed coordinator. It is therefore unclear how accurately its estimates reflect the realized cost-to-go under MSPC. While a formal safety guarantee is not necessary, the paper provides limited evidence that this budget-based screening rule is reliable under the resulting closed-loop policy.

  2. Uncertainty in the reported safety advantage. MSPC achieves encouraging safety coverage, including an improvement over the union of the fixed specialists' safe-task sets. However, its overall advantage over FISOR is relatively small (15/18 versus 14/18 safe tasks), and safety is evaluated by thresholding the mean normalized cost at 1.0. Although the paper reports variability across seeds, it remains unclear how stable the individual constraint-satisfaction outcomes and aggregate safe-task counts are across evaluation configurations. This is particularly relevant for tasks close to the safety threshold. A more explicit account of this uncertainty is needed to assess how robust the reported safety advantage is over the strongest baseline.

Questions for authors
  1. Residual-budget conversion. Could the authors clarify the justification for the discounted budget allowance in Eq. (18)? The conversion appears to scale the remaining episodic budget by the average discount factor, which is not generally equivalent to an undiscounted cumulative-cost constraint when violations occur at different times. Is this intended as a heuristic approximation, and have the authors examined how sensitive the coordinator is to this choice, particularly when safety costs are concentrated late in an episode?

  2. Cost critic reliability under policy coordination. The shared cost critic is learned from the offline dataset using an IQL-style objective, while deployment involves state-dependent switching among multiple specialists. How accurately does the learned critic predict the cost-to-go under the resulting coordinated policy? If available, an analysis comparing predicted and realized costs, particularly for selected actions near the feasibility threshold, would help assess whether estimation errors materially affect the coordinator's decisions.

  3. Robustness of constraint satisfaction. How stable is the reported 15/18 safe-task coverage across the evaluated actor–critic configurations? In particular, do tasks with mean normalized costs close to 1.0 consistently satisfy or violate the constraint across seeds?

  4. Temporal specialization versus general policy diversity. The H8+Global versus Full4 comparison provides useful evidence for the contribution of intermediate horizons. However, adding intermediate horizons also increases the number of candidate policies. Could the authors compare Full4 against an equally sized ensemble of policies trained with the same safety horizon but different random seeds, using the identical coordinator? This would help determine how much of the gain is specific to horizon-dependent safety supervision rather than general policy diversity.

5 comments
AuthorsOpenReview ↗

Reply to W1&Q1: Residual-Budget Conversion and Cost-Timing Sensitivity. (Part 1)

We thank the reviewer for highlighting the sensitivity of Eq. (18) to the temporal distribution of safety costs. The conversion indeed assumes a uniform conditional expected cost rate over the remaining horizon. We adopt this conventional approximation following established safe RL practices (e.g., FAC, CVPO, and CAPS). To further clarify its implications, we provide both an explicit interpretation and empirical comparisons covering early/delayed costs and alternative budget formulations.

1. Assumptions and interpretation. Under a fixed remaining horizon ntn_t and constant conditional expected per-step cost, the discounted and undiscounted expected cumulative costs are related exactly by κγc(nt)=1nt∑j=0nt−1γcj\kappa_{\gamma_c}(n_t)=\frac{1}{n_t}\sum_{j=0}^{n_t-1}\gamma_c^j, yielding the discounted allowance ϵt=btκγc(nt)\epsilon_t=b_t\kappa_{\gamma_c}(n_t). For nonuniform costs, this conversion serves as a time-weighted screening rule, assigning weights wj=γcj/κγc(nt)w_j=\gamma_c^j/\kappa_{\gamma_c}(n_t) to future costs in raw-budget units. Consequently, imminent costs receive greater emphasis, while more distant risks can be reassessed as new observations and budget updates become available. This provides a practical motivation for discounted screening in a feedback-driven coordinator, although delayed costs may be admitted more readily. Importantly, the budget conversion is modular and can be replaced without altering MSPC's core multi-horizon safety specialization.

2. Early/delayed-cost sensitivity. To examine the effect of cost timing under actual deployment, we analyze 2,160 closed-loop episodes across all 18 tasks at B=10B=10, collected from both discounted (γc=0.99\gamma _ {c}=0.99) and undiscounted (γc=1\gamma _ {c}=1) coordinators. Both trained cost critics are cross-evaluated on the same executed state–action pairs and their observed continuations, with equal weighting of the two trajectory sources. We classify observed cost suffixes as early or delayed according to whether more cost occurs in the first or second half of the continuation.

We report two screening diagnostics: pass–unsafe, where an executed action is predicted feasible, but its observed raw-cost continuation exceeds the remaining budget; and reject–safe, where an executed action is predicted infeasible despite its observed continuation remaining within budget. Both rates are measured over all positive-budget coordination queries in each timing group, rather than conditioned on predicted feasibility or infeasibility.

Table: Cost-Timing Sensitivity Under Discounted and Undiscounted Screening. Each entry reports pass–unsafe / reject–safe rates (%).

SuiteTimingγc = 0.99γc = 1
Bullet-Safety-GymEarly0.63 / 48.910.00 / 58.82
Delayed8.74 / 19.876.52 / 50.27
Safety-GymnasiumEarly0.19 / 41.020.00 / 42.19
Delayed3.61 / 49.700.21 / 53.09

For delayed-cost continuations, undiscounted screening reduces pass–unsafe rates from 8.74% to 6.52% on Bullet-Safety-Gym and from 3.61% to 0.21% on Safety-Gymnasium. However, it also increases reject–safe rates, most notably from 19.87% to 50.27% on Bullet-Safety-Gym. These results reveal a screening trade-off: undiscounted screening yields lower pass–unsafe rates for delayed-cost continuations but higher reject–safe rates, while discounted screening classifies more actions whose observed continuations satisfy the remaining budget as feasible.

AuthorsOpenReview ↗

Reply to W1&Q1: Residual-Budget Conversion and Cost-Timing Sensitivity. (Part 2)

3. Undiscounted comparison across deployment budgets. We further examine whether removing discounting improves actual closed-loop performance. Specifically, we retrain the cost critic with γc=1\gamma_c=1 and directly screen candidates against the raw residual budget, ϵt=bt\epsilon_t=b_t, while keeping the temporal specialists and reward critics unchanged. Both formulations are evaluated with frozen checkpoints across B∈5,10,15B\in\\{5,10,15\\} on Bullet and B∈10,15,20B\in\\{10,15,20\\} on Safety-Gymnasium.

As shown in the following table, on Bullet, undiscounted coordination achieves lower mean costs across all three budgets, accompanied by lower rewards. On Safety-Gymnasium, however, the discounted formulation achieves a more favorable trade-off under several settings. At B=10B=10, it obtains both higher reward (0.291 vs. 0.255) and lower mean cost (0.498 vs. 0.607), while maintaining the same safety coverage (8/10). At B=20B=20, it achieves higher reward (0.332 vs. 0.286) and broader safety coverage (10/10 vs. 8/10), despite a slightly higher mean cost (0.399 vs. 0.378).

Table: Discounted versus Undiscounted Coordination Across Deployment Budgets. Reward and Cost/B are suite-level means ± average task-wise standard deviations across actor–critic configurations. Safe tasks indicate the number of tasks with mean Cost/B ≤ 1.

SuiteBudget BγcReward ↑Cost/B ↓Safe Tasks ↑
Bullet-Safety-Gym50.990.331 ± 0.0650.405 ± 0.2826/8
51.000.295 ± 0.0620.368 ± 0.1197/8
100.990.389 ± 0.0720.308 ± 0.2527/8
101.000.314 ± 0.0640.157 ± 0.0628/8
150.990.462 ± 0.0700.398 ± 0.2417/8
151.000.405 ± 0.0600.101 ± 0.0338/8
Safety-Gymnasium100.990.291 ± 0.0280.498 ± 0.2278/10
101.000.255 ± 0.0280.607 ± 0.5028/10
150.990.312 ± 0.0240.435 ± 0.2269/10
151.000.268 ± 0.0290.400 ± 0.3189/10
200.990.332 ± 0.0230.399 ± 0.20210/10
201.000.286 ± 0.0320.378 ± 0.2638/10

Overall summary. These experiments highlight the distinct roles of discounted and undiscounted cost screening. Discounting emphasizes imminent risks and permits greater flexibility for future reassessment, while undiscounted screening places more weight on delayed costs and may exclude otherwise useful actions. The delayed-cost analysis quantifies the corresponding screening trade-off, and the multi-budget results show that removing discounting does not uniformly improve closed-loop reward or safety. Together, these findings support the discounted formulation as a practical operating choice within MSPC, while demonstrating that its multi-horizon coordination framework can also accommodate undiscounted cost estimation.

AuthorsOpenReview ↗

Reply to W1&Q2: Cost-Critic Reliability under Coordination.

We thank the reviewer for raising this important question. Our shared cost critic uses offline IQL-style bootstrapping to provide a consistent, data-supported continuation reference across temporal specialists. We directly assess its prediction accuracy and feasibility-screening reliability under the deployed switching coordinator. This is a deliberate design choice: learning a coordinator-dependent critic would couple value estimation with budget-dependent action selection, as changes in QcQ_c would also change the feasible candidate set and subsequent switching decisions. Instead, we use a shared, data-supported critic to provide a consistent reference for comparing specialist actions. To assess its reliability under actual deployment, we conduct a closed-loop evaluation across all 18 tasks at B=10B=10, comparing predicted costs with realized continuation costs under subsequent specialist switching and budget updates.

We quantify prediction errors through signed bias and MAE against realized discounted continuation costs, and evaluate screening effectiveness using observed raw-budget exceedance rates. As shown in the table below, the shared cost critic exhibits positive mean prediction biases of +5.350 on Bullet-Safety-Gym and +4.067 on Safety-Gymnasium, consistent with a tendency toward conservative cost estimation.

Table: Closed-Loop Cost-Critic Diagnostics at B = 10. Bias and MAE are measured in discounted cost units. Exceedance rates are query-weighted empirical outcomes among predicted-feasible coordination queries with positive residual budgets.

SuiteBiasMAEOverall ExceedanceNear-ThresholdFar-from-Threshold
Bullet-Safety-Gym+5.3505.9635.05%6.51%4.45%
Safety-Gymnasium+4.0674.1920.95%1.61%0.82%

Note: Near- and far-threshold groups are defined by the feasibility margin mt=ϵt−Q^c(st,at)m _ {t}=\epsilon _ {t}-\widehat{Q} _ {c}(s _ {t},a _ {t}) and threshold δt=max⁡(0.1,0.1ϵt)\delta _ {t}=\max(0.1,0.1\epsilon _ {t}). Predicted-feasible queries are near-threshold when 0<mt≤δt0<m _ {t}\leq\delta _ {t} and far-from-threshold when mt>δtm _ {t}>\delta _ {t}.

Among predicted-feasible queries with positive residual budgets, the observed raw-budget exceedance rates are 5.05% on Bullet-Safety-Gym and 0.95% on Safety-Gymnasium. We further examine decisions near the feasibility threshold, where estimation errors can be particularly consequential. Near-threshold exceedance rates are 6.51% and 1.61%, respectively, compared with 4.45% and 0.82% for decisions with larger feasibility margins.

These results provide direct closed-loop evidence supporting the practical reliability of shared-critic screening under budget-dependent policy coordination, with conservative average cost predictions and low observed budget-exceedance rates even near the feasibility boundary.

Reviewer_3ajROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes MSPC for offline safe reinforcement learning. It trains policies using safety supervision over three finite horizons and a recursive Global branch, then coordinates their current action proposals using shared reward/cost critics and the remaining episodic budget. A disagreement gate reduces coordination queries. Experiments investigate performance and temporal complementarity across 18 DSRL tasks.

Critical strengths
  • Retaining policies with different temporal safety supervision is a clear and practically relevant idea.
  • Table 1 reports compliance with the mean-cost budget on 15/18 tasks, compared with 14/18 for FISOR and 13/18 for CAPS-IQL.
  • The fixed-specialist, endpoint-only, and every-step coordination comparisons provide useful evidence about the design choices (Sections 6.1–6.3).
Critical weaknesses
  1. The budget conversion lacks a general justification. In Eq. (18), multiplying the raw residual budget by the average discount factor corresponds to uniform future cost allocation; it does not generally preserve the episodic constraint. For example, with discount 0.99, 100 remaining steps, and budget 10, eleven unit costs at future indices 89–99 have discounted cost approximately 4.279, below the allowance of 6.340, despite totaling 11. Thus, even an exact discounted critic can classify a budget-violating continuation as feasible under Eq. (19). This challenges the screening rule's budget interpretation, although it does not alone establish failure of the complete closed-loop controller.

  2. The cost critic's relationship to the deployed controller is insufficiently established. The IQL targets in Eqs. (15)–(17) do not establish that the critic estimates continuation costs under the actual, budget-dependent switching policy. Taking the maximum of two critics does not by itself establish calibration or an upper bound. Since feasibility decisions depend directly on these predictions, the absence of calibration and false-feasibility measurements leaves a central methodological gap. The paper acknowledges the absence of formal safety guarantees, but empirical validation of this decision mechanism is still needed.

  3. The experiments do not isolate the benefit of horizon diversity. Comparing four specialists with fixed policies or the H8+Global pair also changes the number of action proposals and total model capacity (Table 2). A matched four-policy ensemble using the same horizon and different random seeds is missing. Consequently, the results do not establish how much improvement comes specifically from temporal supervision rather than generic policy diversity and coordination.

  4. The refresh argument does not establish stability of feasibility decisions. Proposition B.4 correctly bounds score differences at a fixed state under a Lipschitz assumption. However, nearby scores can straddle the feasibility threshold, and Algorithm 2 can skip reevaluation while the state and residual budget change. Table 3 supports average efficiency, but does not resolve behavior near budget exhaustion.

Questions for authors
  1. Can Eq. (18) be justified under explicit assumptions or revised? Please evaluate sensitivity to early versus delayed costs and multiple budgets, including comparison with an undiscounted cost formulation.
  2. What continuation policy does the shared cost critic approximate? Please compare its predictions with realized discounted continuation costs under the coordinator, and report how often predicted-feasible decisions lead to violations of the raw residual budget.
  3. Please add a four-policy, same-horizon ensemble with independent seeds, using the identical coordinator and matched capacity and training resources. Does the mixed-horizon family retain a consistent advantage?
  4. Please analyze skipped refreshes near budget exhaustion and compare with a budget-sensitive refresh rule. For the added experiments, separate uncertainty due to actor initialization from critic-seed variation.
5 comments
AuthorsOpenReview ↗

Reply to W1&Q1: Budget Conversion and Empirical Sensitivity. (Part 1)

We thank the reviewer for raising this question. In the following, we clarify the assumption behind Eq.(18) and add both actual closed-loop timing experiments and multi-budget evaluations, including an undiscounted formulation.

1. Explicit assumption and interpretation. Eq. (18) adopts the average-discount-factor conversion used in prior safe RL work, including FAC (2021), CVPO (ICML 2022), and CAPS (AAAI 2025). We provide an explicit derivation under uniform conditional expected per-step costs and further validate its practical behavior under nonuniform cost timing through closed-loop experiments.

Let ntn_t denote the remaining horizon, btb_t the remaining raw budget, and It\mathcal I_t the current conditioning information. Under a constant conditional expected per-step cost, i.e., $\mathbb{E}[c_{t+j}\mid\mathcal I_t]=\mu_t $ for $ j=0,\ldots,n_t-1$, we obtain

E[∑j=0nt−1γcjct+j∣It]=κγc(nt)E[∑j=0nt−1ct+j∣It]\mathbb{E}[\sum _ {j=0} ^ {n _ {t}-1} \gamma _ {c} ^ {j} c _ {t+j} \mid \mathcal{I} _ {t}]=\kappa _ {\gamma _ {c}}(n _ {t})\mathbb{E}[\sum _ {j=0} ^ {n _ {t}-1} c _ {t+j} \mid \mathcal{I} _ {t}]

where

κγc(n)=1n∑j=0n−1γcj.\kappa _ {\gamma _ {c}}(n)=\frac{1}{n}\sum _ {j=0} ^ {n-1}\gamma _ {c} ^ {j}.

Under this assumption, the discounted budget allowance is ϵt=btκγc(nt)\epsilon_t=b_t\kappa_{\gamma_c}(n_t). For nonuniform costs, the conversion instead defines a time-weighted screening rule. Expressed in raw-budget units, future costs receive weights wj=γcjκγc(nt)w_j=\frac{\gamma_c^j}{\kappa_{\gamma_c}(n_t)} placing greater emphasis on imminent risks. This admits a feedback-driven interpretation: near-term risks receive stronger screening, while more distant risks can be reassessed as new observations become available. Importantly, the conversion is not fundamental to MSPC and can be replaced by alternative budget formulations without changing the proposed multi-horizon safety specialization.

2. Early/delayed-cost sensitivity. We analyze actual closed-loop trajectories from both discounted (γc=0.99\gamma_c=0.99) and undiscounted (γc=1\gamma_c=1) coordinators. Both trained critics are cross-evaluated on identical executed state-action samples and observed continuations. The analysis covers all 18 tasks and 2,160 episodes at B=10B=10, with equal weighting of both trajectory sources. Cost suffixes are grouped as early or delayed according to whether more cost occurs in their first or second half.

Table: Cost-timing sensitivity of discounted and undiscounted screening. Pass–unsafe denotes a predicted-feasible decision whose observed raw-cost suffix exceeds the residual budget; reject–safe denotes a predicted-infeasible decision whose observed suffix remains within budget. All entries are percentages of positive-budget query points in the corresponding timing group.

SuiteTimingγ_c = 0.99: Pass–unsafeγ_c = 0.99: Reject–safeγ_c = 1: Pass–unsafeγ_c = 1: Reject–safe
BulletEarly0.6348.910.0058.82
BulletDelayed8.7419.876.5250.27
Safety Gym.Early0.1941.020.0042.19
Safety Gym.Delayed3.6149.700.2153.09

The table above reveals a consistent screening trade-off. Compared with discounted screening, undiscounted screening yields lower pass–unsafe rates among positive-budget query points, particularly for delayed-cost continuations (from 8.74% to 6.52% on Bullet-Safety-Gym and from 3.61% to 0.21% on Safety-Gymnasium), but higher reject–safe rates. This indicates more restrictive screening of future costs, potentially excluding actions whose observed continuations remain within budget. In contrast, discounted screening places greater emphasis on imminent costs while allowing more flexibility for subsequent feedback-driven coordination. This flexibility is accompanied by a higher incidence of pass–unsafe events among positive-budget query points with delayed-cost continuations.

AuthorsOpenReview ↗

Reply to W1&Q1: Budget Conversion and Empirical Sensitivity. (Part 2)

3. Discounted versus undiscounted coordination across budgets. We further compare the original discounted formulation (γc=0.99\gamma_c=0.99) with an undiscounted alternative (γc=1\gamma_c=1) across multiple deployment budgets: B∈5,10,15B\in\\{5,10,15\\} on Bullet and B∈10,15,20B\in\\{10,15,20\\} on Safety-Gymnasium.

For the undiscounted variant, we retrain the cost critic using γc=1\gamma_c=1 and directly compare its predicted costs against the remaining raw budget, ϵt=bt\epsilon_t=b_t. Both variants retain identical temporal specialists and reward critics, while using their respective discount-consistent cost critics and screening thresholds. Each trained cost critic remains frozen across deployment budgets.

Table: Discounted versus Undiscounted Coordination Across Deployment Budgets

Reward and normalized cost (Cost/B) are reported as suite-level means ± average task-wise standard deviations across actor–critic configurations. Safe tasks denote the number of tasks whose mean normalized cost does not exceed 1.0.

SuiteBudget Bγ_cReward ↑Cost/B ↓Safe Tasks ↑
Bullet-Safety-Gym50.990.331 ± 0.0650.405 ± 0.2826/8
51.000.295 ± 0.0620.368 ± 0.1197/8
100.990.389 ± 0.0720.308 ± 0.2527/8
101.000.314 ± 0.0640.157 ± 0.0628/8
150.990.462 ± 0.0700.398 ± 0.2417/8
151.000.405 ± 0.0600.101 ± 0.0338/8
Safety-Gymnasium100.990.291 ± 0.0280.498 ± 0.2278/10
101.000.255 ± 0.0280.607 ± 0.5028/10
150.990.312 ± 0.0240.435 ± 0.2269/10
151.000.268 ± 0.0290.400 ± 0.3189/10
200.990.332 ± 0.0230.399 ± 0.20210/10
201.000.286 ± 0.0320.378 ± 0.2638/10

As shown in the table above, undiscounted coordination achieves lower mean costs across all three deployment budgets on Bullet-Safety-Gym, but consistently yields lower rewards. On Safety-Gymnasium, the results are more nuanced. At B=10, discounted coordination achieves both higher reward (0.291 vs. 0.255) and lower mean cost (0.498 vs. 0.607), with identical safety coverage (8/10). At B=20, discounted coordination also achieves higher reward (0.332 vs. 0.286) and broader safety coverage (10/10 vs. 8/10), although its mean cost is slightly higher (0.399 vs. 0.378).

4. Overall interpretation. These results demonstrate distinct reward–safety trade-offs between discounted and undiscounted coordination. Discounted screening assigns greater weight to imminent costs, while undiscounted screening gives relatively more weight to delayed costs. The timing analysis indicates that undiscounted screening reduces pass–unsafe rates for delayed-cost continuations, but also increases reject–safe rates, potentially excluding actions whose observed continuations remain within budget.

The closed-loop multi-budget results further show that neither formulation uniformly dominates the other. Undiscounted coordination generally achieves lower costs on Bullet-Safety-Gym at the expense of reward, whereas discounted coordination achieves more favorable reward–safety trade-offs in several Safety-Gymnasium settings. These findings support discounted screening as a practical operating choice within the modular MSPC framework, while highlighting the importance of cost timing and task-dependent safety characteristics.

AuthorsOpenReview ↗

Reply to W2&Q2: Continuation Semantics and Closed-Loop Reliability.

We thank the reviewer for highlighting the distinction between the continuation modeled by our shared cost critic and that induced by the deployed switching coordinator. To address this question, we clarify the rationale for using a shared, data-supported continuation reference and provide empirical diagnostics of critic prediction errors and feasibility-screening reliability under actual closed-loop coordination, including near the feasibility boundary.

1. Continuation semantics and design rationale. Our shared cost critic uses IQL-style expectile bootstrapping over offline data to provide a common, data-supported continuation reference across temporal specialists, rather than directly evaluating the deployed budget-dependent switching policy. This decouples critic learning from coordinator decisions, avoiding the feedback coupling that would arise if cost estimates were bootstrapped through the coordinator's own evolving feasibility decisions. We directly validate the resulting critic through prediction-error and feasibility-screening diagnostics on actual closed-loop trajectories.

2. Direct closed-loop validation. We evaluate the frozen MSPC coordinator across all 18 tasks at B=10B=10 and γc=0.99\gamma _ {c}=0.99. At coordination queries with positive residual budgets, we compare the cost critic's predictions for executed actions against their realized discounted continuation costs under subsequent budget-dependent specialist switching and disagreement-triggered refresh. This directly assesses prediction accuracy under the deployed coordinator, using observed trajectories rather than counterfactual outcomes of unexecuted actions.

We report signed prediction bias, mean absolute error (MAE), and the empirical raw-budget exceedance rate among predicted-feasible queries:

Pr⁡emp(Gtraw>bt∣Q^c(st,at)<ϵt,;query,;bt>0).\Pr _ {\mathrm{emp}}(G _ {t} ^ {\mathrm{raw}}>b _ {t}\mid\widehat{Q} _ {c}(s _ {t},a _ {t})<\epsilon _ {t},\\;\mathrm{query},\\;b _ {t}>0).

To examine reliability near the feasibility boundary, we define the screening margin mt=ϵt−Q^c(st,at)m _ {t}=\epsilon _ {t}-\widehat{Q} _ {c}(s _ {t},a _ {t}) and threshold δt=max⁡(0.1,0.1ϵt)\delta _ {t}=\max(0.1,0.1\epsilon _ {t}). Predicted-feasible queries are classified as near-threshold when 0<mt≤δt0<m _ {t}\leq\delta _ {t} and far-from-threshold when mt>δtm _ {t}>\delta _ {t}. Future costs are used exclusively for post-hoc evaluation, with all trained actors and critics kept frozen.

Table: Closed-Loop Cost-Critic Diagnostics at B = 10. Bias and MAE are measured in discounted cost units, averaged within configurations and then equally across configurations and tasks. Budget-exceedance rates are query-weighted empirical outcomes among predicted-feasible queries with positive residual budgets.

SuiteBiasMAEOverall ExceedanceNear-ThresholdFar-from-Threshold
Bullet-Safety-Gym+5.3505.9635.05% (2,240/44,344)6.51% (847/13,010)4.45% (1,393/31,334)
Safety-Gymnasium+4.0674.1920.95% (1,150/121,399)1.61% (308/19,133)0.82% (842/102,266)

As shown in the table above, the shared cost critic exhibits positive mean prediction biases of +5.350 on Bullet-Safety-Gym and +4.067 on Safety-Gymnasium, consistent with conservative cost estimation. We further evaluate the effectiveness of this screening mechanism through empirical raw-budget exceedance rates, particularly for decisions near the feasibility threshold.

Among predicted-feasible queries with positive residual budgets, observed raw-budget exceedance rates are 5.05% on Bullet-Safety-Gym and 0.95% on Safety-Gymnasium, supporting empirical screening effectiveness. Even near the feasibility threshold, the exceedance rates remain limited to 6.51% and 1.61%, decreasing further to 4.45% and 0.82% for decisions with larger feasibility margins. These results provide additional empirical evidence of effective cost-based screening under actual closed-loop coordination.

Together, the conservative average cost predictions, low empirical budget-exceedance rates, and consistent near/far-threshold risk separation provide direct closed-loop evidence supporting the practical reliability of shared-critic screening under the deployed MSPC coordinator.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.