acceptodds
Under review as a conference paper at ICLR 2027

Stochastic Decision Horizons for Survival-Based Constrained Reinforcement Learning

Abstract

We propose stochastic decision horizons (SDH), a theoretically grounded framework for survival-based constrained reinforcement learning from per-step violation signals. Rather than treating violations as expenditure of a cumulative budget, SDH shortens the effective decision horizon when violations occur, reducing current reward credit and future bootstrapping. We show that this construction represents a geometric-horizon survival constraint and, for a fixed Lagrange multiplier, remains the same SDH problem up to a constant reward offset that acts as an implicit incentive for survival. Using control as inference, we derive two off-policy and replay-compatible, regularized algorithms with post-violation semantics. Absorbing-state ends the decision process, so only surviving decisions pay policy KL, yielding AS-SAC. Virtual termination stops reward credit while the process continues, retaining ordinary discounted KL and yielding VT-MPO. To connect SDH with cumulative-cost CMDPs, we introduce violation-depth profiles. Under exponential-tail single-scale profiles, the SDH statistic determines cumulative cost; departures from this form mark where the connection breaks. Experiments validate both the method and this predicted scope. On the 90-muscle H2190 humanoid in Hyfydy, VT-MPO matches the EWA reference in peak gait realism, with its highest observed gait-match checkpoint at 21M environment steps versus 90M for EWA and markedly more stable training. On Safety Gymnasium, violation-depth profiles predict the regimes in which SDH achieves strong reward-violation trade-offs.

Primary area:
reinforcement learning
Keywords:
constrained reinforcement learning, chance-constrained reinforcement learning, survival-based reinforcement learning, control as inference, off-policy reinforcement learning, state-action dependent discounting, continuous control, musculoskeletal control

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2Y6JOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors introduce Stochastic Decision Horizons (SDH). It is a new framework for constrained reinforcement learning (RL) that handles safety violations through survival probabilities instead of cumulative-cost constraints.

Critical strengths
  1. A strong and principled mathematical foundation is represented by the geometric-horizon interpretation, survival-weighted Bellman equation, and fixed-multiplier Lagrangian formulation.

  2. Evaluation is performed on H2190 humanoid with 90 muscles, which provides a challenging continuous-control setting.

Critical weaknesses
  1. The Hyfydy experiments use only three random seeds, which is relatively small for assessing the reliability of the proposed methods.
Questions for authors

Please refer to the weakness section.

2 comments
AuthorsOpenReview ↗

Response to Reviwer 2Y6J

We thank the reviewer for recognizing the strength of the geometric-horizon formulation, the survival-weighted Bellman theory, and the ambitious 90-muscle H2190 demonstration. We have directly addressed the reliability concern by expanding the main Hyfydy evaluation to five independent seeds per method, with 100M environment steps each (Sec. 5.1; Fig. 3; Table 6). The expanded five-seed evaluation, together with the additional experiments and theoretical refinements summarized below, is already incorporated into the updated manuscript. Revised discussions are highlighted in green for ease of review.

More robust gait evaluation. Every 200k steps we evaluate ten deterministic rollouts per seed; each method's best-observed checkpoint is selected using the across-seed mean at a common training step. The updated gait matches (mean ±\pm SD) of VT-MPO vs. EWA are 75.1±2.2%75.1\pm2.2\% vs. 72.8±5.4%72.8\pm5.4\% (H0918), 69.5±2.1%69.5\pm2.1\% vs. 68.5±3.3%68.5\pm3.3\% (H1622), and 62.6±3.2%62.6\pm3.2\% vs. 59.2±1.9%59.2\pm1.9\% (H2190). On H2190, the selected peak occurs at 18M steps for VT-MPO versus 76M for EWA. Human gait match is an independent kinematic metric that neither algorithm directly optimizes; these results show that a fixed SDH control objective produces natural walking comparable to a specialized, adaptively weighted reference.

Further additions addressing the other reviews. The revision substantially extends the evidence supporting the framework. A new CaT-PPO comparison, using the same effort reward and eight violation signals, finds stable gait in 15/15 VT-MPO seeds versus 0/15 CaT-PPO seeds across H0918, H1622, and H2190 (Fig. 4; Table 8); a warm-start test reveals the importance of feasible-policy discovery (Table 9). New adaptive-η\eta experiments explicitly target survival probabilities ρ∈{0.9,0.7}\rho\in\{0.9,0.7\}, achieving five of eight target settings and revealing controlled reward-survival trade-offs (Sec. 5.2; Tables 1, 16). A direct primal-dual comparison at the benchmark's native additive-cost budget D=25D=25 evaluates the SDH survival Lagrangian against a standard episodic-cost Lagrangian baseline under matched training and evaluation conditions (Table 17). Both approaches achieve budget-feasible mean costs on all four velocity tasks, while SDH obtains higher returns on every task. This provides direct empirical evidence for the effectiveness of survival-based constrained optimization.

We also strengthen the violation-depth theory with a simple probabilistic interpretation: pπ=Pr⁡π(GTγ>0)p_\pi=\Pr_\pi(G_{T_\gamma}>0) measures violation incidence, while τπ\tau_\pi captures conditional violation severity. Under the single-scale exponential assumption, Pr⁡π(GTγ>b∣GTγ>0)=e−b/τπ\Pr_\pi(G_{T_\gamma}>b\mid G_{T_\gamma}>0)=e^{-b/\tau_\pi}, the violation-depth profile becomes Ωπ(b)=pπe−b/τπ\Omega_\pi(b)=p_\pi e^{-b/\tau_\pi}, yielding the exact identity Jc(π)=pπτπJ_c(\pi)=p_\pi\tau_\pi. This formulation naturally accommodates violation-free trajectories and establishes a precise connection between survival statistics and cumulative cost (Sec. 2.2; Theorem F.20).

I thank the authors for their detailed response and the additional experiments incorporated into the revised manuscript. The expanded evaluation with five independent seeds, together with the additional gait comparisons, CaT-PPO baseline, adaptive survival-probability experiments, and primal-dual comparison, strengthens the empirical evidence supporting the proposed framework.

I appreciate the effort taken to address the reviewers' concerns and believe that the revision has improved the overall quality and completeness of the paper. However, while increasing the number of random seeds from three to five is a positive step, the evaluation still provides relatively limited evidence for establishing the statistical robustness and reproducibility of the proposed approach, particularly in challenging continuous-control settings.

Nevertheless, considering the theoretical contributions, the additional experimental evidence, and the improvements made in the revision, I find the paper suitable for acceptance. I therefore maintain my original rating of 3 (Weak Accept).

Reviewer_VofTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

In many control tasks, the system must respect limits at every step: a simulated walker, for example, must stay within ranges of speed, ground-reaction force, joint torque, and muscle use. Standard constrained RL instead accumulates costs over time and keeps the total under a budget. This suits resources such as fuel, but not limits that must hold at every step, where behaving well for a while does not "earn" a later violation.

The paper proposes Stochastic Decision Horizons (SDH). Whenever the agent violates a limit, SDH reduces both the reward at that step and the weight of future rewards, with larger violations causing stronger reductions. Violations thus shorten the agent's effective planning horizon. The environment remains unchanged; only the learning target is modified.

The paper makes two main contributions:

  1. A constrained-problem interpretation and two algorithms. The authors show that shrinking the horizon solves a constrained problem: maximizing reward earned while "alive", subject to a minimum survival probability ρ\rho (e.g., 90%). Since reward and constraints share the same survival process, a single Lagrange multiplier suffices regardless of the number of combined limits, acting as a constant per-step survival bonus. With a reward of "one minus effort", the constant one already serves this purpose, eliminating multiplier tuning. The authors derive two off-policy algorithms differing in whether policy regularization stops after a violation: AS-SAC stops, matching tasks that terminate upon violation, while VT-MPO continues, matching systems that remain operational. AS-SAC achieves higher rewards but is more seed-sensitive, whereas VT-MPO is more conservative and stable, making it the recommended default.

  2. The violation-depth profile describes how long a policy spends above each accumulated-violation level. Standard cumulative-cost constraints limit the area under this curve, whereas SDH captures only a weighted summary. The authors show that SDH cannot generally replace a cumulative budget, but the two coincide when the profile follows a single exponential shape.

Validation. On three simulated musculoskeletal humanoids (up to 90 muscles; 3 seeds), VT-MPO achieves walking performance comparable to the specialized EWA method (62.9% vs. 64.6% gait match on the largest model), without multiplier updates, with smoother training and an earlier best checkpoint (21M vs. 90M steps). On eight Safety Gymnasium tasks (5 seeds), compared with standard constrained RL baselines, violation-depth profiles help explain when SDH performs well and when cumulative budgets remain necessary.

Critical strengths

Strength 1

A demanding, high-dimensional application solved with a fixed objective. The Hyfydy experiments recast EWA's heuristic effort-weight adaptation as a constrained problem with physically interpretable limits (velocity, ground-reaction force, joint torque, muscle sparsity) and solve it with a fixed objective (η=0\eta=0, fixed λ\lambda ramp). On three musculoskeletal models, including the 90-muscle H2190, VT-MPO matches the domain-specific state of the art in gait realism (a metric neither method optimizes), keeps the operating limits, and trains noticeably more stably than EWA and than tuned PID-Lagrangian baselines. This provides clear evidence that the formulation is useful in practice as it replaces non-stationary reward-weight or dual schedules with interpretable thresholds.

Strength 2

A coherent account of why continuation/termination-based constrained RL works. Theorem 2.2 makes precise what termination-style methods optimize and explains why they often work without dual updates: a positive constant reward is only collected while "alive", so it acts as a survival incentive. The paper is also explicit about scope, with clean counterexamples showing that SDH is not a general additive-budget CMDP solver. This gives practitioners who already use termination tricks a principled interpretation, and states clearly what SDH does not guarantee.

Critical weaknesses

Weakness 1: Limited theoretical and empirical support for violation-depth profiles

The violation-depth profiles are insufficiently supported, both theoretically and empirically. The theory assumes that every trajectory has experienced a violation by a random cutoff time, excluding safe policies with many violation-free trajectories. Moreover, the profiles are computed from unconstrained SAC policies rather than the SDH policies being evaluated.

The experimental results do not consistently align with the profiles:

  • SDH satisfies the cost limit on several tasks labeled "Mismatch".
  • The highlighted failures occur on tasks where nearly all methods fail.
  • Some labels are inconsistent between the table and the text.

Furthermore, the experiments do not test the theorem's condition: SDH uses a fixed survival bonus and a predefined schedule rather than the survival threshold linked to the cost limit. Consequently, the claim that these profiles can predict when SDH succeeds is not sufficiently supported.

Weakness 2: Insufficient isolation of the first contribution

The features distinguishing the first contribution from prior work are not isolated empirically. In particular:

  • Constrained formulation: All experiments use a fixed η\eta (0 on Hyfydy, 0.1 on Safety Gym) and a hand-set λ\lambda schedule. No experiment targets a survival probability ρ\rho through primal-dual updates (listed as future work). Thus, SDH is evaluated primarily as a fixed reward-shaping scheme rather than a constrained RL solver.
  • AS/VT distinction: The comparison is confounded with the underlying algorithms: AS is only paired with SAC and VT with MPO. Crossed variants are not evaluated, and the authors themselves attribute the aggressive-versus-stable behavior to known differences between SAC and MPO.
  • VT-MPO implementation: MPO's KL constraint is averaged over replay states without time weighting. In practice, VT-MPO therefore resembles MPO with an SDH critic, making the claim that this is "not an approximation" questionable.
  • AS ablation: Removing the "living cost" has only a minor effect on performance, raising questions about its practical contribution.

Weakness 3: Insufficient discussion of closely related work

The novelty relative to the closest prior work remains unclear. Termination-as-constraint methods [1, 2] receive only a brief mention and no empirical comparison, although CaT [1] appears to be a particularly relevant baseline.

Several related research directions are also missing from the discussion:

  • Viability and safe-value-function methods [3, 4], showing that failure penalties or terminations can encourage safe behavior without dual updates.
  • Chance-constrained RL [5, 6] and safety critics [7].
  • Discount-as-survival interpretations [8, 9].
  • Survival incentives induced by positive rewards under termination [10].
  • Inference-based constrained RL [11].

Without a more thorough discussion and comparison, it is difficult to distinguish the paper's novel contributions from established results reformulated in terms of survival.

References

[1] Chane-Sane, E., Leziart, P. A., Flayols, T., Stasse, O., Souères, P., & Mansard, N. (2024). Cat: Constraints as terminations for legged locomotion reinforcement learning. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 13303-13310). IEEE.

[2] Sun, H., Xu, Z., Peng, Z., Fang, M., Wang, T., Dai, B., & Zhou, B. (2022). Constrained mdps can be solved by eearly-termination with recurrent models. In NeurIPS 2022 Foundation Models for Decision Making Workshop.

[3] Massiani, P. F., Heim, S., Solowjow, F., & Trimpe, S. (2022). Safe value functions. IEEE Transactions on Automatic Control, 68(5), 2743-2757.

[4] Aubin, J. P., Bayen, A. M., & Saint-Pierre, P. (2011). Viability theory: new directions (Vol. 2). Berlin: Springer.

[5] Chow, Y., Ghavamzadeh, M., Janson, L., & Pavone, M. (2018). Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research, 18(167), 1-51.

[6] Paternain, S., Calvo-Fullana, M., Chamon, L. F., & Ribeiro, A. (2022). Safe policies for reinforcement learning via primal-dual methods. IEEE Transactions on Automatic Control, 68(3), 1321-1336.

[7] Srinivasan, K., Eysenbach, B., Ha, S., Tan, J., & Finn, C. (2020). Learning to be safe: Deep rl with a safety critic. arXiv preprint arXiv:2010.14603.

[8] PPitis, S. (2019, July). Rethinking the discount factor in reinforcement learning: A decision theoretic approach. In Proceedings of the AAAI conference on artificial intelligence (Vol. 33, No. 01, pp. 7949-7956).

[9] Fedus, W., Gelada, C., Bengio, Y., Bellemare, M. G., & Larochelle, H. (2019). Hyperbolic discounting and learning over multiple horizons. arXiv preprint arXiv:1902.06865.

[10] Li, A., Misra, D., Kolobov, A., & Cheng, C. A. (2023). Survival instinct in offline reinforcement learning. Advances in neural information processing systems, 36, 62062-62120.

[11] Liu, Z., Cen, Z., Isenbaev, V., Liu, W., Wu, S., Li, B., & Zhao, D. (2022, June). Constrained variational policy optimization for safe reinforcement learning. In International Conference on Machine Learning (pp. 13644-13668). PMLR.

Questions for authors

Question 1

Can SDH reach a specified survival level ρ\rho with primal-dual updates on η\eta, as the theory proposes? How does it then compare with Lagrangian CMDP baselines tuned to the same violation level, and with CaT using the same violation signals? A positive answer would substantially strengthen the claim that SDH is a constrained RL method rather than a fixed shaping scheme.

Question 2

Can you report the crossed variants AS-MPO and VT-SAC, at least on Hyfydy H1622 and a few Safety Gym tasks? Without them, it is not possible to tell whether any observed difference comes from the AS/VT semantics or from SAC vs. MPO.

Additional questions (if needed)

  1. (Clarification.) Why are off-policy, replay-compatible algorithms needed to realize the SDH objective? The paper presents this as a contribution but does not justify it or compare against an on-policy variant.

Presentation The paper is hard to read. Notation is very dense (Tγ,Bt,Ct,Atσ,ut,vtσ,μαπ,Sˉπ,Sπ,ΩπT_\gamma, B_t, C_t, A^\sigma_t, u_t, v^\sigma_t, \mu^\pi_\alpha, \bar S_\pi, \mathfrak S_\pi, \Omega_\pi; both κ\kappa and β\beta as temperatures), basic objects such as rr, γ\gamma, CMDP, and gait quality are used before they are defined, key terms are stacked into long noun phrases (e.g., "survival-gated geometric-horizon chance-constrained problem"), and heavy hedging makes the actual claims hard to extract. Much of the needed material is in a ~50-page appendix. A short plain-language statement of the problem, a worked example of the (r~,γ~)(\tilde r,\tilde\gamma) transformation, and a notation table would help considerably.

2 comments
AuthorsOpenReview ↗

Response to Reviewer VofT

We sincerely thank the reviewer for the thoughtful and detailed feedback. In particular, the observation that our original violation-depth theory excluded violation-free trajectories motivated a deeper probabilistic analysis. The resulting refinement separates violation incidence from conditional severity. All revised discussions are highlighted in green for ease of review, with precise references to the relevant sections, figures, and tables below.

1. Violation depth: corrected theory and direct tests. The revised theory separates violation incidence pπ=Pr⁡π(GTγ>0)p_\pi=\Pr_\pi(G_{T_\gamma}>0) from conditional severity τπ\tau_\pi. Under the positive-depth model Pr⁡π(GTγ>b∣GTγ>0)=e−b/τπ\Pr_\pi(G_{T_\gamma}>b\mid G_{T_\gamma}>0)=e^{-b/\tau_\pi}, we derive Ωπ(b)=pπe−b/τπ\Omega_\pi(b)=p_\pi e^{-b/\tau_\pi}, Jc(π)=pπτπJ_c(\pi)=p_\pi\tau_\pi, and Sˉπ(λ)=1−pπ+pπ/(1+λτπ)\bar S_\pi(\lambda)=1-p_\pi+p_\pi/(1+\lambda\tau_\pi) (Sec. 2.2, Eq. 6; Theorem F.20). Mass 1−pπ1-p_\pi at zero admits violation-free policies; the identities show when survival statistics and incidence recover cumulative cost.

App. E.3.4 now evaluates trained SDH and Lagrangian policies, using five seeds, 100 test rollouts per seed, scale-consistency tests, and held-out cost reconstruction (Tables 19–20). On RacecarCircle1, the policies have similar violation-free episode shares (0.800.80 vs. 0.770.77), but geometric-horizon costs differ (0.420.42 vs. 3.093.09), as do positive-depth scales (1313 vs. 2727). This isolates the importance of violation severity. Profile shape diagnoses the survival-cost relation; observed costs establish feasibility.

2. Primal-dual optimization and controlled comparisons. We implement adaptive-η\eta VT-MPO to target explicit finite-episode survival probabilities: the probability of completing an episode without constraint violations must be at least ρ∈{0.9,0.7}\rho\in\{0.9,0.7\}. Experiments use 5M transitions and five seeds (Sec. 5.2; Tables 1, 16). Five of eight settings meet their target. PointGoal1 attains 0.94±0.010.94\pm0.01 and 0.73±0.020.73\pm0.02; relaxing the target increases return from 0.30.3 to 9.99.9. PointPush1 attains 0.73±0.010.73\pm0.01 for ρ=0.7\rho=0.7 (return 5.25.2), and approaches the stricter 0.90.9 target at 0.88±0.010.88\pm0.01. CarButton1 remains challenging.

A second experiment directly compares primal-dual optimization under the SDH survival Lagrangian against a standard episodic-cost Lagrangian baseline at the benchmark's native budget D=25D=25 (Table 17; Fig. 5A). Both use the same underlying learner, replay pipeline, five seeds, and 5M transitions. Both achieve budget-feasible mean costs on all four velocity tasks, while SDH obtains consistently higher returns: Ant 16951695 vs. 671671, HalfCheetah 22652265 vs. 13271327, Hopper 14041404 vs. 468468, Humanoid 42554255 vs. 35283528. Compared with fixed η\eta, adaptive VT-MPO improves return on 7/87/8 tasks and lowers cost on 6/86/8 (Table 18). In Hyfydy, the exact identity (1−γ)μαπ(1−f)=psurv(π)−(1−γ)μαπ(f)(1-\gamma)\mu^\pi_\alpha(1-f)=p_{\mathrm{surv}}(\pi)-(1-\gamma)\mu^\pi_\alpha(f) explains why r=1−effortr=1-\mathrm{effort} already contains a direct survival objective at η=0\eta=0 (Table 7).

3. AS/VT is a mathematical distinction, independent of optimizer choice. Theorem 3.1 derives distinct CaI KL weights: vtAS=utv_t^{\mathrm{AS}}=u_t shares survival weighting with reward, whereas vtVT=γtv_t^{\mathrm{VT}}=\gamma^t retains ordinary discounting (Eq. 8). AS admits a variable-discount soft Bellman recursion (AS-SAC); VT pairs the survival critic with MPO improvement (Secs. 3–4). Crossed AS-MPO/VT-SAC experiments would assess alternative implementations; the distinct objectives follow directly from the derivation. We present observed stability differences as comparisons of the implemented algorithms.

4. Replay and AS living cost. VT-MPO changes MPO's critic to the SDH Bellman target r~η=α(r+η)\tilde r_\eta=\alpha(r+\eta), γ~=γα\tilde\gamma=\gamma\alpha, while retaining the original MPO E/M steps and replay-state KL averaging (Eqs. 11–12; App. B). Raw reward/violation data in replay let us recompute targets under the current λ,η\lambda,\eta. Under AS, the survival-weighted living cost is policy dependent (Corollary H.1); removing it changes optimal continuation from 0.7070.707 to 0.9840.984 in Counterexample H.3 and changes Ant mean return from 2779±642779\pm64 to 1895±8381895\pm838 in the ablation (App. E.6, Table 23).

5. CaT and presentation. With identical effort reward and eight violation signals, VT-MPO discovers stable walking in 15/15 seeds versus 0/15 for CaT-PPO over 100M steps (Fig. 4; App. D.3.2, Table 8). CaT-PPO with DEP also fails from scratch, while actor-only BC warm starts retain 984984–10001000-step H0918 gaits (Table 9). This identifies discovery of viable policies as the practical challenge. Sec. 1 now covers termination, viability, chance constraints, safety critics, and inference-based methods; Table 2 collects notation.

Reviewer_VofTOpenReview ↗

Main concerns addressed, with one remainingqQuestion about AS/VT stability

I thank the authors for the detailed response and additional experiments. The revised violation-depth analysis now accounts for violation-free trajectories and evaluates trained policies directly. The adaptive VT-MPO experiments also address my main empirical concern by showing that the method can meet prescribed survival targets in five of eight settings. I appreciate the added CaT comparison and the clearer positioning relative to prior work.

My remaining concern is the interpretation of the stability differences between AS-SAC and VT-MPO. The derivation establishes that AS and VT optimize distinct objectives, but the experiments still leave open how much of the observed difference comes from the objectives versus the choice of SAC or MPO. Overall, the revision addresses my main concerns, and I will raise my score accordingly.

16:29

Reviewer_QdCsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes stochastic decision horizons (SDH), which map per-step violations to a continuation probability scaling both immediate rewards and future discounting. It formulates a geometric-horizon survival constraint and shows that its fixed-multiplier Lagrangian is an SDH objective with a constant reward shift. A control-as-inference analysis distinguishes absorbing-state and virtual-termination regularization, yielding AS-SAC and VT-MPO. Violation-depth profiles establish when survival statistics determine cumulative costs, under a restrictive exponential-tail assumption.

Validation covers three musculoskeletal humanoids and eight Safety Gymnasium tasks. The locomotion results are strongest: VT-MPO approximately matches EWA’s peak gait realism with smoother training; its best observed checkpoint on the 90-muscle H2190 occurs at 21M versus 90M steps. Safety Gymnasium results are mixed: CarButton remains problematic, whereas AS-SAC becomes feasible on PointGoal with longer training.

Critical strengths

The musculoskeletal results demonstrate practical value on a demanding control problem. The paper is careful about its scope, and the AS/VT analysis usefully clarifies policy regularization.

Critical weaknesses

Limited incremental novelty. CaT already uses the basic reward/discount transformation. The advance therefore rests largely on the theoretical interpretation and off-policy extensions, without a direct comparison establishing their practical benefit over this predecessor.

Constraint enforcement remains unvalidated. All experiments fix (\eta); none tests achieving a prescribed survival probability. The exact Lagrangian identity does not itself establish constraint satisfaction.

Restricted empirical generality. Locomotion thresholds are extensively tuned against gait-match, with peak checkpoints selected using only three seeds. Violation-depth profiles are measured under unconstrained policies, limiting the evidence for their predictive validity.

Questions for authors

The paper’s central premise is that some limits are best treated as “local viability” conditions rather than as a budget to be spent. In a real application, how should a practitioner decide which kind of limit they face, before training anything?

3 comments
AuthorsOpenReview ↗

Response to Reviewer QdCs

We appreciate the reviewer's recognition of the demanding musculoskeletal results and the principled AS/VT analysis. The revision directly addresses the three empirical questions raised: comparison with CaT, prescribed survival levels, and reliability beyond the original evaluation. All new experiments, theoretical refinements, and supporting results discussed below are already incorporated into the updated manuscript. Revised discussions are highlighted in green for ease of review.

1. What SDH adds beyond CaT. Continuation is the mechanism; our contribution is to establish the constrained optimization problem it solves and derive practical off-policy algorithms from it. SDH couples survival-weighted task reward and survival probability at the same geometric evaluation time. Theorem 2.2 gives the exact fixed-multiplier identity L(π,η)=(1−γ)μαπ(r+η)−ηρ\mathcal{L}(\pi,\eta)=(1-\gamma)\mu^\pi_\alpha(r+\eta)-\eta\rho. Thus a single η\eta controls a survival constraint built from arbitrarily many local violation signals, and the reward shift r↦r+ηr\mapsto r+\eta directly incentivizes survival. Crucially, unlike standard discounted RL, where adding a constant reward does not change the optimal policy, under SDH a unit reward contributes exactly the survival probability: (1−γ)μαπ(1)=psurv(π)(1-\gamma)\mu^\pi_\alpha(1)=p_{\mathrm{surv}}(\pi). This yields our key insight for Hyfydy: with r=1−effortr=1-\mathrm{effort}, the objective already contains a direct survival incentive, enabling effective optimization with fixed η=0\eta=0, without dual updates (Sec. 5.1; Table 7).

2. Direct closest-prior comparison. We implemented CaT-PPO on the same three Hyfydy humanoids, effort reward, and eight violation signals (Fig. 4; App. D.3.2, Table 8). Across 100M steps, VT-MPO discovers stable walking in 15/15 seeds; CaT-PPO does so in 0/15. The final episode lengths are 1000±01000\pm0, 963±7963\pm7, and 980±7980\pm7 for VT-MPO, versus 48±148\pm1, 37±137\pm1, and 30±130\pm1 for CaT-PPO. Adding DEP exploration to CaT-PPO does not resolve gait acquisition. When initialized by behavior cloning a VT-MPO H0918 actor, CaT-PPO maintains 984984–10001000-step deterministic gaits after 30M additional steps (Table 9). The gap is therefore strongly associated with finding viable policies from scratch.

3. Survival-probability targeting and budget comparisons. We now train adaptive-η\eta VT-MPO to satisfy explicit finite-episode survival requirements: the probability that an entire episode contains no constraint violations must be at least ρ∈{0.9,0.7}\rho\in\{0.9,0.7\}. We evaluate five seeds over 5M transitions (Sec. 5.2; Tables 1, 16). Five of eight task-target configurations reach their requested level. On PointGoal1, the measured survival shares are 0.94±0.010.94\pm0.01 and 0.73±0.020.73\pm0.02; changing the target from 0.90.9 to 0.70.7 raises return from 0.30.3 to 9.99.9. PointPush1 similarly increases return from 1.61.6 to 5.25.2 while meeting the 0.70.7 target. We also report the more difficult PointPush1 0.90.9 target (0.88±0.010.88\pm0.01) and CarButton1. These experiments demonstrate direct feedback control of a chosen survival operating point, complementing the geometric-horizon theory.

For the benchmark's native additive budget D=25D=25, we directly compare primal-dual optimization under the SDH survival Lagrangian against a standard episodic-cost Lagrangian baseline (Table 17; Fig. 5A). Both approaches use the same underlying learner, replay pipeline, five-seed evaluation, and 5M transitions. Both achieve budget-feasible mean costs on all four velocity tasks, while SDH obtains higher returns on every task, e.g. 16951695 vs. 671671 (Ant), 22652265 vs. 13271327 (HalfCheetah), and 14041404 vs. 468468 (Hopper).

4. Reliability and when to use survival. We expanded the principal Hyfydy comparison to five seeds with ten deterministic rollouts per seed every 200k steps (Sec. 5.1, Fig. 3, Table 6). On the 90-muscle H2190, peak observed gait match is 62.6±3.2%62.6\pm3.2\% for VT-MPO versus 59.2±1.9%59.2\pm1.9\% for EWA, at selected checkpoints of 18M and 76M steps. Gait match is an external human-kinematics metric optimized by neither method. The physically interpretable operating limits are calibrated in advance (App. D.2); training then uses a fixed objective, in contrast with EWA's changing reward weights.

The practitioner decision is physical: use an additive budget for resources transferable across time (e.g. energy), and survival for operating regimes whose quality depends on each local decision (e.g. gait velocity, joint torque, and ground-reaction forces). Our revised violation-depth theory makes the distinction quantitative: violation incidence pπp_\pi and conditional depth scale τπ\tau_\pi characterize when survival statistics recover cumulative cost (Sec. 2.2; Theorem F.20). New diagnostics evaluate trained constrained policies, with five seeds and 100 test rollouts per seed (App. E.3.4, Tables 19–20).

I thank the authors for a substantial revision. Two concerns nevertheless remain.

  1. The CaT comparison does not isolate the contribution of SDH. The authors accept that continuation is the mechanism shared with CaT. By App. B, VT-MPO is standard MPO with a continuation-scaled critic. CaT-PPO and VT-MPO therefore differ chiefly in the learner: on-policy PPO versus off-policy MPO. Adding DEP to CaT-PPO does not change this, since the learner remains PPO. The authors attribute the 0/15 vs 15/15 gap to discovering viable policies from scratch, but this design cannot separate that effect from the change of learner. An informative test would hold the learner fixed, e.g. CaT’s termination probabilities inside the same MPO/DEP pipeline.

  2. The survival-targeting experiments do not test the constraint the theory characterises. They target the probability of a violation-free episode. Theorem 2.2 instead concerns the geometric-horizon survival Pr(B_{T_γ} = 1), which under exponential continuation equals E[e^{−λG_{T_γ}}]. The rebuttal itself describes these experiments as “complementing” the theory.

They are also reported only on Safety Gymnasium, meet the target in 5 of 8 settings, and on PointGoal1 meet ρ = 0.9 with a return of only 0.3. Meanwhile, the main Hyfydy application still uses fixed η = 0. The native-budget comparison is reported only for the four velocity tasks, on which SDH was already feasible in the original submission.

I maintain my score.

AuthorsOpenReview ↗

Response to Reviewer QdCs

We thank the reviewer for the constructive feedback and suggestions, which motivated the additional controlled CaT experiment. We have updated the manuscript with these results.

1. VT-MPO with CaT Continuation

We performed the requested fixed-learner ablation, replacing SDH's exponential continuation with CaT's termination-probability rule while keeping the VT-MPO/DEP pipeline, architectures, replay, exploration, reward, and eight operating-limit signals fixed (Fig. 4b; App. D.3.2, E4).

Both rules discover stable walking in 15/15 seeds across the three Hyfydy models. However, SDH reaches the 900-step threshold 2.3-5.3 times earlier in median environment steps (H0918: 2.4M vs. 8.6M; H1622: 10.2M vs. 23.6M; H2190: 10.0M vs. 53.0M). It also shows fewer post-acquisition drops below 500 steps (0/15 vs. 5/15) and lower mean aggregate operating-limit violations across all models.

This addresses the reviewer's requested control: CaT continuation supports gait discovery under VT-MPO, so CaT-PPO's failure (0/15) cannot be attributed to continuation alone. Actor-only warm starts further show that CaT-PPO can retain an initialized walking policy, pointing to a feasible-policy discovery bottleneck in the tested PPO pipeline. Importantly, with the learner fixed, our exponential continuation rule still improves learning efficiency, stability, and average limit adherence. We do not attribute the PPO-MPO gap specifically to on-policy sampling or claim to isolate individual components of the continuation rules.

2. Why Fixed η=0\eta=0 Provides a Survival Incentive

Using fixed η=0\eta=0 in Hyfydy is a principled choice, not an omission of survival optimization. Theorem 2.2 shows that, unlike standard discounted RL, a constant reward under SDH directly incentivizes survival. For r=1−fr=1-f, we obtain (1−γ)μαπ(1−f)=Pr⁡π(BTγ=1)−(1−γ)μαπ(f).(1-\gamma)\mu_\alpha^\pi(1-f) =\Pr_\pi(B_{T_\gamma}=1)-(1-\gamma)\mu_\alpha^\pi(f). Thus, even at η=0\eta=0, the objective explicitly balances geometric-horizon survival probability against effort, without an additional multiplier.

We tested this prediction on H1622 by comparing the original objective against survival-only optimization (r=1r=1), changing only the VT-MPO policy-improvement reward (Sec. 5.1, App. D.3.1, Table 7). Survival-only optimization did not improve survival: 0.683±0.0140.683\pm0.014 versus 0.691±0.0070.691\pm0.007 for the original objective (r=1−effortr=1-\mathrm{effort}). The original also achieved higher task reward (0.8220.822 vs. 0.7940.794) and lower aggregate per-step violations (0.08160.0816 vs. 0.11140.1114).

Together, these results support our choice of η=0\eta=0 for Hyfydy. The effort-aware SDH objective already provides a direct survival incentive, while retaining effort information yields higher task reward and lower operating-limit violations without sacrificing observed survival. This supports using a simple, fixed objective for Hyfydy, without introducing additional multiplier adaptation.

3. Safety Gymnasium: Testing the Scope of SDH

As in all Lagrangian baselines of SafePO/OmniSafe, whose critics optimize discounted costs while the multiplier tracks the undiscounted episodic constraint, our primal step optimizes the exact SDH Lagrangian of Theorem 2.2 for each η\eta, while the dual step tracks the finite-episode survival requirement a practitioner specifies. On rollouts, the geometric-horizon survival Pr⁡(BTγ=1)\Pr(B_{T_\gamma}=1) of the same policies is 0.970.97–0.990.99 at γ=0.99\gamma=0.99, so the finite-episode target is the more demanding specification.

The results show both potential and limits. Five of eight survival targets are met; PointGoal1 reaches ρ=0.9\rho=0.9 at return 0.30.3, while CarButton remains difficult. On four velocity tasks, the matched native-budget comparison achieves feasible mean costs and higher returns than an episodic Lagrangian baseline.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Revisions and Key Clarifications

The reviewers recognize two central strengths of SDH: a principled survival-constrained interpretation of continuation-based RL and convincing high-dimensional musculoskeletal control, including a 90-muscle humanoid. We have used the discussion to strengthen the same core contribution through a more general and interpretable violation-depth theory, direct comparisons with the closest prior method, explicit survival-constraint optimization, and broader empirical validation. Importantly, all these advances are already incorporated into our updated submission. The manuscript contains the strengthened theoretical analysis, all newly conducted experiments, and their complete results and evaluation protocols in the main text and appendix. All revised discussions are highlighted in green for ease of review, with precise references provided below.

A complete constrained-RL account. Theorem 2.2 establishes the geometric-horizon survival constraint and its exact Lagrangian, L(π,η)=(1−γ)μαπ(r+η)−ηρ\mathcal L(\pi,\eta)=(1-\gamma)\mu^\pi_\alpha(r+\eta)-\eta\rho. Unlike standard RL, a constant reward shift under SDH directly incentivizes survival. This explains why r=1−effortr=1-\mathrm{effort} already provides a survival incentive in Hyfydy, enabling effective optimization with fixed η=0\eta=0 (Sec. 5.1; Table 7). Theorem 3.1 derives the distinct AS/VT regularization objectives underlying the replay-compatible AS-SAC and VT-MPO algorithms. The strengthened violation-depth theory (Sec. 2.2; Theorem F.20) separates violation incidence from severity, accommodates violation-free trajectories, and characterizes when survival statistics recover additive cost, now examined on learned constrained policies (App. E.3.4).

New decisive experiments. First, the requested direct CaT-PPO baseline uses the same effort reward and eight local violation signals: VT-MPO learns stable walking in 15/15 seeds, versus 0/15 for CaT-PPO at 100M steps (Fig. 4a; Table 8). We additionally performed the requested fixed-learner ablation, replacing our exponential continuation with CaT's rule inside the same VT-MPO/DEP pipeline. Both rules discover walking in 15/15 seeds, but our exponential rule reaches the 900-step threshold 2.3-5.3 times earlier, with fewer post-acquisition drops below 500 steps (0/15 vs. 5/15) and lower mean aggregate operating-limit violations (Fig. 4b; App. D.3.2, E4). Warm-start diagnostics show that CaT-PPO can maintain a gait discovered by VT-MPO (Table 9), illuminating the difficulty of policy discovery. Together, these results show that CaT continuation supports gait discovery, while the training pipeline matters for discovering viable policies and continuation design affects learning efficiency, stability, and limit adherence.

Reliability and presentation. We expanded the main Hyfydy comparison to five seeds. On H2190, VT-MPO reaches 62.6±3.2%62.6\pm3.2\% peak observed gait match versus 59.2±1.9%59.2\pm1.9\% for EWA, with selected checkpoints at 18M versus 76M steps (Sec. 5.1, Table 6). The revised paper also strengthens the CaT and safety literature discussion, clarifies the precise AS/VT contribution and practical replay update, and adds a notation table (Sec. 1; Table 2).

Together, the revisions establish SDH as a survival-based constrained control framework with a precise objective, principled off-policy algorithms, and substantial evidence on both high-dimensional locomotion and specified safety targets. They directly address the major questions raised while preserving the paper's original theoretical contributions and focus on challenging real-world control problems.

Area_Chair_ojjkOpenReview ↗

Reviewer–Author Discussion Period Ending Soon

Dear Reviewers,

With the discussion period coming to a close, I encourage you to revisit the authors’ responses and let us know if you have any further comments.

Thank you for your careful reviews and engagement in the discussion!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.