Weakness 1: Limited theoretical and empirical support for violation-depth profiles
The violation-depth profiles are insufficiently supported, both theoretically and empirically. The theory assumes that every trajectory has experienced a violation by a random cutoff time, excluding safe policies with many violation-free trajectories. Moreover, the profiles are computed from unconstrained SAC policies rather than the SDH policies being evaluated.
The experimental results do not consistently align with the profiles:
- SDH satisfies the cost limit on several tasks labeled "Mismatch".
- The highlighted failures occur on tasks where nearly all methods fail.
- Some labels are inconsistent between the table and the text.
Furthermore, the experiments do not test the theorem's condition: SDH uses a fixed survival bonus and a predefined schedule rather than the survival threshold linked to the cost limit. Consequently, the claim that these profiles can predict when SDH succeeds is not sufficiently supported.
Weakness 2: Insufficient isolation of the first contribution
The features distinguishing the first contribution from prior work are not isolated empirically. In particular:
- Constrained formulation: All experiments use a fixed η (0 on Hyfydy, 0.1 on Safety Gym) and a hand-set λ schedule. No experiment targets a survival probability ρ through primal-dual updates (listed as future work). Thus, SDH is evaluated primarily as a fixed reward-shaping scheme rather than a constrained RL solver.
- AS/VT distinction: The comparison is confounded with the underlying algorithms: AS is only paired with SAC and VT with MPO. Crossed variants are not evaluated, and the authors themselves attribute the aggressive-versus-stable behavior to known differences between SAC and MPO.
- VT-MPO implementation: MPO's KL constraint is averaged over replay states without time weighting. In practice, VT-MPO therefore resembles MPO with an SDH critic, making the claim that this is "not an approximation" questionable.
- AS ablation: Removing the "living cost" has only a minor effect on performance, raising questions about its practical contribution.
Weakness 3: Insufficient discussion of closely related work
The novelty relative to the closest prior work remains unclear. Termination-as-constraint methods [1, 2] receive only a brief mention and no empirical comparison, although CaT [1] appears to be a particularly relevant baseline.
Several related research directions are also missing from the discussion:
- Viability and safe-value-function methods [3, 4], showing that failure penalties or terminations can encourage safe behavior without dual updates.
- Chance-constrained RL [5, 6] and safety critics [7].
- Discount-as-survival interpretations [8, 9].
- Survival incentives induced by positive rewards under termination [10].
- Inference-based constrained RL [11].
Without a more thorough discussion and comparison, it is difficult to distinguish the paper's novel contributions from established results reformulated in terms of survival.
References
[1] Chane-Sane, E., Leziart, P. A., Flayols, T., Stasse, O., Souères, P., & Mansard, N. (2024). Cat: Constraints as terminations for legged locomotion reinforcement learning. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 13303-13310). IEEE.
[2] Sun, H., Xu, Z., Peng, Z., Fang, M., Wang, T., Dai, B., & Zhou, B. (2022). Constrained mdps can be solved by eearly-termination with recurrent models. In NeurIPS 2022 Foundation Models for Decision Making Workshop.
[3] Massiani, P. F., Heim, S., Solowjow, F., & Trimpe, S. (2022). Safe value functions. IEEE Transactions on Automatic Control, 68(5), 2743-2757.
[4] Aubin, J. P., Bayen, A. M., & Saint-Pierre, P. (2011). Viability theory: new directions (Vol. 2). Berlin: Springer.
[5] Chow, Y., Ghavamzadeh, M., Janson, L., & Pavone, M. (2018). Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research, 18(167), 1-51.
[6] Paternain, S., Calvo-Fullana, M., Chamon, L. F., & Ribeiro, A. (2022). Safe policies for reinforcement learning via primal-dual methods. IEEE Transactions on Automatic Control, 68(3), 1321-1336.
[7] Srinivasan, K., Eysenbach, B., Ha, S., Tan, J., & Finn, C. (2020). Learning to be safe: Deep rl with a safety critic. arXiv preprint arXiv:2010.14603.
[8] PPitis, S. (2019, July). Rethinking the discount factor in reinforcement learning: A decision theoretic approach. In Proceedings of the AAAI conference on artificial intelligence (Vol. 33, No. 01, pp. 7949-7956).
[9] Fedus, W., Gelada, C., Bengio, Y., Bellemare, M. G., & Larochelle, H. (2019). Hyperbolic discounting and learning over multiple horizons. arXiv preprint arXiv:1902.06865.
[10] Li, A., Misra, D., Kolobov, A., & Cheng, C. A. (2023). Survival instinct in offline reinforcement learning. Advances in neural information processing systems, 36, 62062-62120.
[11] Liu, Z., Cen, Z., Isenbaev, V., Liu, W., Wu, S., Li, B., & Zhao, D. (2022, June). Constrained variational policy optimization for safe reinforcement learning. In International Conference on Machine Learning (pp. 13644-13668). PMLR.
Response to Reviwer 2Y6J
We thank the reviewer for recognizing the strength of the geometric-horizon formulation, the survival-weighted Bellman theory, and the ambitious 90-muscle H2190 demonstration. We have directly addressed the reliability concern by expanding the main Hyfydy evaluation to five independent seeds per method, with 100M environment steps each (Sec. 5.1; Fig. 3; Table 6). The expanded five-seed evaluation, together with the additional experiments and theoretical refinements summarized below, is already incorporated into the updated manuscript. Revised discussions are highlighted in green for ease of review.
More robust gait evaluation. Every 200k steps we evaluate ten deterministic rollouts per seed; each method's best-observed checkpoint is selected using the across-seed mean at a common training step. The updated gait matches (mean ± SD) of VT-MPO vs. EWA are 75.1±2.2% vs. 72.8±5.4% (H0918), 69.5±2.1% vs. 68.5±3.3% (H1622), and 62.6±3.2% vs. 59.2±1.9% (H2190). On H2190, the selected peak occurs at 18M steps for VT-MPO versus 76M for EWA. Human gait match is an independent kinematic metric that neither algorithm directly optimizes; these results show that a fixed SDH control objective produces natural walking comparable to a specialized, adaptively weighted reference.
Further additions addressing the other reviews. The revision substantially extends the evidence supporting the framework. A new CaT-PPO comparison, using the same effort reward and eight violation signals, finds stable gait in 15/15 VT-MPO seeds versus 0/15 CaT-PPO seeds across H0918, H1622, and H2190 (Fig. 4; Table 8); a warm-start test reveals the importance of feasible-policy discovery (Table 9). New adaptive-η experiments explicitly target survival probabilities ρ∈{0.9,0.7}, achieving five of eight target settings and revealing controlled reward-survival trade-offs (Sec. 5.2; Tables 1, 16). A direct primal-dual comparison at the benchmark's native additive-cost budget D=25 evaluates the SDH survival Lagrangian against a standard episodic-cost Lagrangian baseline under matched training and evaluation conditions (Table 17). Both approaches achieve budget-feasible mean costs on all four velocity tasks, while SDH obtains higher returns on every task. This provides direct empirical evidence for the effectiveness of survival-based constrained optimization.
We also strengthen the violation-depth theory with a simple probabilistic interpretation: pπ=Prπ(GTγ>0) measures violation incidence, while τπ captures conditional violation severity. Under the single-scale exponential assumption, Prπ(GTγ>b∣GTγ>0)=e−b/τπ, the violation-depth profile becomes Ωπ(b)=pπe−b/τπ, yielding the exact identity Jc(π)=pπτπ. This formulation naturally accommodates violation-free trajectories and establishes a precise connection between survival statistics and cumulative cost (Sec. 2.2; Theorem F.20).