- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies contrastive reinforcement learning (CRL / Scaling‑CRL) for goal‑conditioned control under failure‑terminated MDPs, where fall or collision immediately terminates episodes. The authors identify a systematic over‑estimation bias originating from per‑trajectory normalization during hindsight relabeling: short near‑failure trajectories get artificially amplified gradient weight, leading to two empirical failure modes: catastrophic failure bootstrapping and unsustainable goal‑reaching (agent briefly touches goal but fails immediately afterwards). To fix this bias, the authors propose Safe‑CRL with two complementary corrections: mass‑weighted InfoNCE for critic training and a log‑survival‑mass score for actor optimization, plus a lightweight Z‑encoder trained purely from one‑bit failure termination signals.
Critical strengths
Interesting theoretical identification of a specific bias inside CRL‑InfoNCE framework. The work points out that per‑trajectory normalization on pre‑failure futures under absorbing failure termination creates survival‑mass distortion. The bias is intrinsic to CRL’s contrastive formulation rather than general hindsight‑relabeling or standard TD over‑estimation. Theoretical propositions and proofs are provided in appendix, decomposing bias into critic‑side distribution distortion and missing survival‑mass factor for actor ranking.
Critical weaknesses
- Missing comparison against closely‑related SOTA baseline SVL / Survival‑RL. SVL (Survival Reinforcement Learning) targets nearly identical problem setting: self‑supervised goal‑conditioned learning under failure‑terminated episodes. Although SVL is not built upon InfoNCE‑CRL, it is the most relevant contemporary work. The paper only compares against Scaling‑CRL and puts limited SAC‑HER results into appendix, without evaluating SVL on key representative environments. Without this comparison, it is hard to judge whether the observed improvement is competitive against alternative survival‑aware self‑supervised RL paradigms.
- Simulation environments are modified and not community‑standard benchmarks. The Humanoid locomotion tasks intentionally weaken actuator gears and reduce ground friction to produce abundant early failure‑terminated trajectories, which magnifies the survival‑mass bias effect. Therefore reported results cannot be directly compared against numbers from existing literature using original Brax dynamics. It remains unclear how large this bias would be under standard, unmodified Humanoid settings. There is no ablation sweeping over environment failure rates to quantify bias magnitude as early‑failure frequency changes.
- Evaluation metric compatibility issue hinders cross‑work reference. The core metric time‑at‑goal is custom‑adopted in this paper and is not widely used by safe‑RL community. The paper does not report conventional standard metrics (e.g. raw goal success rate, episodic return) alongside the proposed metrics, making cross‑paper reference difficult.
Questions for authors
Questions To Authors Q1. Could you add SVL / Survival‑RL baseline on several representative environments such as Humanoid Goal and Ant Big Pitfall for direct comparison? Q2. What is the performance of Safe‑CRL on original unmodified Brax Humanoid dynamics without weakened actuators? How large is the survival‑mass bias effect under standard failure rates? Please consider ablating over environment failure probability. Q3. Can you report conventional metrics (goal success rate, mean episodic return) together with time‑at‑goal for easier cross‑paper referencing? Q4. Could you add a heuristic ablation: simple filtering of early‑failed episodes, to compare against your loss‑based correction? Q5. How well‑calibrated are Z‑encoder predictions quantitatively, beyond the qualitative trend plot?
2 comments
Remind of reviewer-author discussion
Dear reviewer,
The discussion period is ending soon. We would appreciate it if you could kindly share your thoughts and feedback on our responses. We will be glad to provide any further clarifications if needed. Thank you.
Best regards, Authors
Responses to Reviewer EEAB
We thank the reviewer for the constructive and insightful feedback. Our responses to the raised questions are given below. Revisions in the updated manuscript are in blue text.
Q1 Additional baselines
We did cite SVL and Survival RL in the related work section. SVL studies offline RL with given demonstrations. The SVL setting is quite different from our online RL tasks. Survival RL (SRL) is indeed relevant to our method. But SRL aims to solve only the problem of how to stay at the goal safely. Obstacle avoidance and the mitigation of catastrophic failure bootstrapping are not considered.
In the revised manuscript, we compare Scaling-CRL, Safe-CRL, and Survival RL on 4 representative tasks: Point Goal, Ant Goal, Ant Big Pitfall, and Hum. Goal. The result is presented in the added Figure 8, page 8. As expected, Survival RL is competitive only on Ant Goal, the only task without obstacles and with a stable robot. Its 'dwell-time' objective promotes stability after goal-arrival but cannot avoid obstacles and falls as Safe-CRL does. This is an inherent flaw of Survival RL.
Note that Survival RL was still on arXiv when we submitted our paper, and its complete repo was released on October 4 during the ICLR 2027 review period. The ICLR 2027 reviewer guidelines do not require comparison with such contemporaneous work. Nevertheless, we hope the added experiments address this concern.
Q2 Failure-rate ablation / original CRL environment
Safe-CRL targets failure-prone tasks, so adding objects and slippery ground to amplify the bias is a reasonable choice. When failures are rare, logZ approaches 0 and Safe-CRL becomes identical to the original CRL.
We realize that our description of weakened actuator gears was misleading. MuJoCo has multiple backends; the original CRL uses
springwith stronger default actuators. This backend is no longer maintained by Brax and fails on many newer Hopper/Blackwell GPUs with atcache 2memory-management error. We therefore use the recommended, actively maintainedmjxbackend. Its recent version already has weaker default humanoid gears and more frequent failures thanspring, as discussed in the original CRL appendices (Wang et al., 2025). We now added clarifications in Appendix B, pages 18/19.We also added an experiment on Humanoid Goal and Humanoid U-Maze using the original
springbackend on V100 GPUs. The results use 3 seeds and 64-layer networks and are included in Table D.2, page 22 (copied below, final performance). Safe-CRL retains a significant advantage on humanoid tasks.However, a controlled failure-rate sweep remains difficult to configure. Determining failure-rate or survival-time curves requires complete training runs. We cannot finish this ablation during rebuttal. But we believe that the additional results with the original spring backend can address your concern.
Q3 Conventional metrics
Our reward is +1 at the goal and 0 otherwise (Section 2.1). Addressing this sparse reward motivates goal-conditioned RL. Thus, time at goal equals episodic goal-reaching return; survival time equals the return from +1 per surviving step; and goal coverage is the goal-reaching success rate (%). We do not use dense rewards such as progress toward the goal.
Q4 Early-failure filtering ablation
We added this ablation as required in Section 6; see Figure 10, page 9. Filtered CRL's performance is poor: most early trajectories end in failure, so filtering them out dilutes the replay buffer and substantially reduces sample efficiency.
Q5 Quantitative Z-encoder calibration
All 3 reviewers asked about the treatment of time-limit truncation. Please see our shared response to Q1 of Reviewer 1 (8y2k). The added calibration results in Figure 6, page 7 show that the learned Z is accurate.
We hope that our responses can address the raised concerns and the reviewer can re-evaluate the contribution of our paper.