acceptodds
Under review as a conference paper at ICLR 2027

Arrive and Survive: Scaling Safe Goal-Conditioned Policy Learning from One-Bit Failure Signals

Abstract

Contrastive reinforcement learning (CRL) scales effectively in goal-conditioned tasks through self-supervision. However, under failure termination, established CRL considers pre-failure future goals only when constructing positive samples, discarding the occupancy mass absorbed by failure. This omission induces a systematic overestimation bias in the implied goal-reaching values, giving short surviving futures disproportionately strong supervision. Such distortion can reinforce failure-prone behaviour through catastrophic failure bootstrapping or favour unsustainable goal reaching. To address this problem, we introduce two simple yet theoretically rigorous corrections: mass-weighted InfoNCE corrects the overweighting of short surviving futures in critic learning, and a log-survival-mass score restores the missing mass in policy optimization under the population formulation. The resulting method, Safe Contrastive Reinforcement Learning (Safe-CRL), uses the one-bit failure termination signal as its only safety supervision. Across twelve failure-prone robot navigation and locomotion tasks, Safe-CRL consistently improves mean survival time and achieves higher mean time at goal than Scaling-CRL, with substantial gains in nine tasks. It also preserves depth scalability, with deep policies exhibiting complex failure-avoidance behaviours. These results establish survival-mass correction as an effective completion of CRL for failure-prone goal-conditioned control.

Primary area:
reinforcement learning
Keywords:
contrastive reinforcement learning, goal-conditioned reinforcement learning, safe reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_EEABOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies contrastive reinforcement learning (CRL / Scaling‑CRL) for goal‑conditioned control under failure‑terminated MDPs, where fall or collision immediately terminates episodes. The authors identify a systematic over‑estimation bias originating from per‑trajectory normalization during hindsight relabeling: short near‑failure trajectories get artificially amplified gradient weight, leading to two empirical failure modes: catastrophic failure bootstrapping and unsustainable goal‑reaching (agent briefly touches goal but fails immediately afterwards). To fix this bias, the authors propose Safe‑CRL with two complementary corrections: mass‑weighted InfoNCE for critic training and a log‑survival‑mass score for actor optimization, plus a lightweight Z‑encoder trained purely from one‑bit failure termination signals.

Critical strengths

Interesting theoretical identification of a specific bias inside CRL‑InfoNCE framework. The work points out that per‑trajectory normalization on pre‑failure futures under absorbing failure termination creates survival‑mass distortion. The bias is intrinsic to CRL’s contrastive formulation rather than general hindsight‑relabeling or standard TD over‑estimation. Theoretical propositions and proofs are provided in appendix, decomposing bias into critic‑side distribution distortion and missing survival‑mass factor for actor ranking.

Critical weaknesses
  1. Missing comparison against closely‑related SOTA baseline SVL / Survival‑RL. SVL (Survival Reinforcement Learning) targets nearly identical problem setting: self‑supervised goal‑conditioned learning under failure‑terminated episodes. Although SVL is not built upon InfoNCE‑CRL, it is the most relevant contemporary work. The paper only compares against Scaling‑CRL and puts limited SAC‑HER results into appendix, without evaluating SVL on key representative environments. Without this comparison, it is hard to judge whether the observed improvement is competitive against alternative survival‑aware self‑supervised RL paradigms.
  2. Simulation environments are modified and not community‑standard benchmarks. The Humanoid locomotion tasks intentionally weaken actuator gears and reduce ground friction to produce abundant early failure‑terminated trajectories, which magnifies the survival‑mass bias effect. Therefore reported results cannot be directly compared against numbers from existing literature using original Brax dynamics. It remains unclear how large this bias would be under standard, unmodified Humanoid settings. There is no ablation sweeping over environment failure rates to quantify bias magnitude as early‑failure frequency changes.
  3. Evaluation metric compatibility issue hinders cross‑work reference. The core metric time‑at‑goal is custom‑adopted in this paper and is not widely used by safe‑RL community. The paper does not report conventional standard metrics (e.g. raw goal success rate, episodic return) alongside the proposed metrics, making cross‑paper reference difficult.
Questions for authors

Questions To Authors Q1. Could you add SVL / Survival‑RL baseline on several representative environments such as Humanoid Goal and Ant Big Pitfall for direct comparison? Q2. What is the performance of Safe‑CRL on original unmodified Brax Humanoid dynamics without weakened actuators? How large is the survival‑mass bias effect under standard failure rates? Please consider ablating over environment failure probability. Q3. Can you report conventional metrics (goal success rate, mean episodic return) together with time‑at‑goal for easier cross‑paper referencing? Q4. Could you add a heuristic ablation: simple filtering of early‑failed episodes, to compare against your loss‑based correction? Q5. How well‑calibrated are Z‑encoder predictions quantitatively, beyond the qualitative trend plot?

2 comments
AuthorsOpenReview ↗

Responses to Reviewer EEAB

We thank the reviewer for the constructive and insightful feedback. Our responses to the raised questions are given below. Revisions in the updated manuscript are in blue text.

Q1 Additional baselines

We did cite SVL and Survival RL in the related work section. SVL studies offline RL with given demonstrations. The SVL setting is quite different from our online RL tasks. Survival RL (SRL) is indeed relevant to our method. But SRL aims to solve only the problem of how to stay at the goal safely. Obstacle avoidance and the mitigation of catastrophic failure bootstrapping are not considered.

In the revised manuscript, we compare Scaling-CRL, Safe-CRL, and Survival RL on 4 representative tasks: Point Goal, Ant Goal, Ant Big Pitfall, and Hum. Goal. The result is presented in the added Figure 8, page 8. As expected, Survival RL is competitive only on Ant Goal, the only task without obstacles and with a stable robot. Its 'dwell-time' objective promotes stability after goal-arrival but cannot avoid obstacles and falls as Safe-CRL does. This is an inherent flaw of Survival RL.

Note that Survival RL was still on arXiv when we submitted our paper, and its complete repo was released on October 4 during the ICLR 2027 review period. The ICLR 2027 reviewer guidelines do not require comparison with such contemporaneous work. Nevertheless, we hope the added experiments address this concern.

Q2 Failure-rate ablation / original CRL environment

Safe-CRL targets failure-prone tasks, so adding objects and slippery ground to amplify the bias is a reasonable choice. When failures are rare, log⁡Z\log Z approaches 0 and Safe-CRL becomes identical to the original CRL.

We realize that our description of weakened actuator gears was misleading. MuJoCo has multiple backends; the original CRL uses spring with stronger default actuators. This backend is no longer maintained by Brax and fails on many newer Hopper/Blackwell GPUs with a tcache 2 memory-management error. We therefore use the recommended, actively maintained mjx backend. Its recent version already has weaker default humanoid gears and more frequent failures than spring, as discussed in the original CRL appendices (Wang et al., 2025). We now added clarifications in Appendix B, pages 18/19.

We also added an experiment on Humanoid Goal and Humanoid U-Maze using the original spring backend on V100 GPUs. The results use 3 seeds and 64-layer networks and are included in Table D.2, page 22 (copied below, final performance). Safe-CRL retains a significant advantage on humanoid tasks.

Env. (steps)MethodTime at goalSurvival timeGoal coverage (%)
Hum. Goal (100M)Scaling-CRL523.71 ± 112.38762.3 ± 141.789.3 ± 1.1
Safe-CRL688.54 ± 33.71901.8 ± 15.992.5 ± 0.7
Hum. U-Maze (200M)Scaling-CRL55.64 ± 44.91311.7 ± 111.038.8 ± 2.2
Safe-CRL212.33 ± 120.05503.2 ± 188.463.9 ± 1.6

However, a controlled failure-rate sweep remains difficult to configure. Determining failure-rate or survival-time curves requires complete training runs. We cannot finish this ablation during rebuttal. But we believe that the additional results with the original spring backend can address your concern.

Q3 Conventional metrics

Our reward is +1 at the goal and 0 otherwise (Section 2.1). Addressing this sparse reward motivates goal-conditioned RL. Thus, time at goal equals episodic goal-reaching return; survival time equals the return from +1 per surviving step; and goal coverage is the goal-reaching success rate (%). We do not use dense rewards such as progress toward the goal.

Q4 Early-failure filtering ablation

We added this ablation as required in Section 6; see Figure 10, page 9. Filtered CRL's performance is poor: most early trajectories end in failure, so filtering them out dilutes the replay buffer and substantially reduces sample efficiency.

Q5 Quantitative Z-encoder calibration

All 3 reviewers asked about the treatment of time-limit truncation. Please see our shared response to Q1 of Reviewer 1 (8y2k). The added calibration results in Figure 6, page 7 show that the learned Z is accurate.

We hope that our responses can address the raised concerns and the reviewer can re-evaluate the contribution of our paper.

AuthorsOpenReview ↗

Remind of reviewer-author discussion

Dear reviewer,

The discussion period is ending soon. We would appreciate it if you could kindly share your thoughts and feedback on our responses. We will be glad to provide any further clarifications if needed. Thank you.

Best regards, Authors

Reviewer_1LLROpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper shows that, under failure termination, CRL's hindsight relabelling samples positive goals only from pre-failure states and renormalizes each trajectory to unit mass. This discards the discounted occupancy absorbed by failure ("survival mass" Z) and over-rewards trajectories that reach the goal shortly before failing. The authors propose Safe-CRL, with two corrections: a mass-weighted InfoNCE that weights each row by the realized survival mass (fixing the critic's normalization), and a log Z term, predicted by a small Z-encoder, added to the actor's score (restoring the missing mass). The only safety signal is the one-bit termination flag. On 12 navigation and locomotion tasks, Safe-CRL improves survival time and time at goal over Scaling-CRL on all tasks, with large gains on nine, preserves depth scaling, and adds about 2.8% training time. Ablations show the log Z term drives most of the gain.

Critical strengths

Strength 1

A clear, new diagnosis with a principled, minimal fix. The paper identifies a specific way CRL's Monte Carlo relabelling loses the termination signal that TD methods retain, and Figure 1 illustrates the resulting behaviour well. The correction is derived from the factorization Q = Z · µ̄ rather than by tuning, which explains why a survival term belongs in the actor, in what form, and with what default weight. It needs no safety labels or hand-designed costs, only the termination signal already available, and leaves the rest of Scaling-CRL unchanged, so it is easy to adopt in existing CRL code.

Strength 2

Large gains at negligible cost, with depth scaling preserved and a fairly thorough study. Survival time improves on all 12 tasks and time at goal improves substantially on nine, most strikingly on Humanoid tasks (e.g., Humanoid Big Pitfall time at goal 8.2 → 474.1). The overhead is about 4% parameters and 2.8% wall-clock, and 8-layer Safe-CRL often matches 64-layer Scaling-CRL. The main results use 5 seeds, and the ablations (both components, the log Z weight, Z-encoder depth, TD-based Z, goal respawning) include results that do not favour the method, which adds credibility.

Critical weaknesses

Weakness 1

The guarantee covers an idealized version of the method. Corollary 1 is proved for the calibrated score f̄ + log Z, but the actor is trained on the raw score f + log Z (Eq. 18). The authors transparently note (Sec. 3.2, A.3) that calibration is an analysis device and that Safe-CRL "retains Scaling-CRL's treatment of the contrastive-score offset," but this treatment is not specified. Since the InfoNCE offset c(s, a) can vary across actions, it could distort or swamp the log Z correction. Relatedly, the critic weights (A.5) treat time-limit truncation like failure, while the Z-encoder treats it as survival, so the two corrections use inconsistent notions of survival near episode ends. Both gaps weaken the "theoretically exact" framing.

Weakness 2

The evaluation cannot show whether the specific correction matters, rather than any use of the failure signal. Only one baseline is compared. Scaling-CRL was run in environments the authors made more failure-prone, apparently without retuning, and it essentially fails on the Humanoid tasks (e.g., 0.56 time at goal on Humanoid U-Maze). The ablation shows that "log Z only" (uncorrected critic plus log Z), which the theory does not justify, recovers nearly all the gains (Table 2), and β = 5 often beats the derived β = 1 on survival (Fig. 8). Since Z(s, a) is goal-independent, log Z acts like a learned stay-alive bonus, yet there is no comparison with other simple ways of using the same one-bit signal: a tuned survival bonus or failure penalty, failure as an explicit absorbing outcome (argued against in Sec. 3.2 but not tested), survival-analysis methods such as SRL/SVL (cited but not compared, with no reason given), or a Lagrangian/safety-critic method treating termination as a cost. In addition, time at goal rewards exactly the post-arrival stability that log Z targets, the failure rate is not reported, goal coverage drops on five of six Ant tasks (e.g., Ant Cross Maze 56.1 → 46.0) without mention in the abstract, and the qualitative rollouts (Fig. 7) show no failure cases.

Questions for authors

Question 1

What exactly is "Scaling-CRL's treatment of the contrastive-score offset" (e.g., a logsumexp regularizer)? Can you show that the residual action-dependent offset is small relative to log Z, or report a variant with explicit calibration in the actor objective?

Question 2

(Additional baselines.) How does Safe-CRL compare with at least one alternative using the same one-bit failure signal: Scaling-CRL with a tuned survival bonus or failure penalty, CRL with failure as an explicit absorbing outcome, a survival-analysis method such as SRL/SVL (or an explanation of why this comparison is not feasible), or a Lagrangian/safety-critic method treating termination as a cost? Clearly beating these would show that the derived form and weight matter, which would address our main concern.

Additional questions

Q3. What fraction of anchors are down-weighted due to time-limit truncation rather than failure, and how do results change if they are excluded or reweighted?

Q4. Were Scaling-CRL's hyperparameters re-tuned for the modified dynamics (reduced friction, weakened Humanoid gears), given near-zero baseline results on several Humanoid tasks?

Q5. Scaling-CRL's Car Goal survival is 464.9 in both Table 1 (5 seeds, final policy, ± 6.5) and Table 2 (3 seeds, final 10% of training, ± 11.2). Are these from the same runs?

Q6. Catastrophic failure bootstrapping and unsustainable goal reaching are presented as consequences of the bias but only shown to co-occur with it. Can the link be supported more directly, e.g., with an example where trajectory normalization flips the action ranking? Is the bias specific to CRL or does it apply to hindsight relabelling in general?

4 comments
AuthorsOpenReview ↗

Responses to Reviewer 1LLR [part 1]

We thank the reviewer for the constructive feedback. The raised questions are answered below. Revisions in the updated manuscript are in blue text.

Q1 Score offset regularization and actor calibration

Safe-CRL retains the logsumexp (LSE) regularizer in CRL methods. Eysenbach et al. (2022) proposed this regularization; Bortkiewicz et al. (2025) further validate its importance for CRL (Section 5.3 of that paper). The LSE penalty encourages c(s,a)c(s,a) to approach a constant −log⁡B-\log B, where BB is the batch size. As a constant does not provide a gradient, comparing its value with ZZ is not necessary.

As LSE regularization is an established and verified mechanism in CRL, we prefer not to revalidate it again in this paper, but to add the necessary clarifications:

  • Section 3.2, page 5: "Following established CRL (Eysenbach et al., 2022; Bortkiewicz et al., 2025), we retain the squared LSE regularizer to constrain the anchor-dependent InfoNCE offset. No additional actor-side calibration is induced."

  • Appendix A.3, pages 15-16: "The inherited LSE penalty (Eysenbach et al., 2022) constrains the anchor-dependent offset. At the population density-ratio optimum, it encourages a common action-independent offset, approximately −log⁡B-\log B (BB is batch size), making raw scores comparable across actions."

Q2 Additional baselines

We added 2 additional baselines/ablations on 4 representative tasks:

1. Survival RL (SRL): SVL addresses offline RL without failure termination and is incompatible with our online RL setting. So, we added the Survival RL baseline. Please see our response to Q1 of Reviewer 3 (EEAB) for details. The result is shown in Figure 8, page 8, with added discussion. In short, SRL aims to make the agent stay at the goal safely. It cannot handle obstacle avoidance and early failures as effectively as Safe-CRL. This is an inherent design flaw of SRL.

2. Survival bonus baseline: We added a baseline, CRL+(Z-1), which replaces log⁡Z\log Z with Z−1Z-1. Their values and gradients are identical as Z→1Z\rightarrow1; for Z<1Z<1, Z−1Z-1 has a smaller gradient but a higher score. The results are shown in Figure 10, page 9. Across 4 environments, the difference is marginal on Point and Ant tasks, whose initial survival times and Z values are higher. Humanoid starts with a survival time near 0, so the larger gradient of log⁡Z\log Z encourages faster, safer learning. This partly explains Safe-CRL's greater advantage on humanoid tasks.

Theoretically, log⁡Z\log Z restores the missing survival mass while retaining goal occupancy as the sole objective (Corollary 1). An arbitrary reward bonus or a coefficient other than 1 introduces another objective, which may align or conflict with goal occupancy. Accordingly, Safe-CRL performs consistently well across all settings (Figures 9 and 10), whereas other methods perform well on some tasks but poorly on others. The added Figure 10 and the existing Figure 9 together justify the claim that using the logarithm term is critical to maintaining the goal occupancy objective.

3. Other baselines: (1) Lagrangian/safety-critic methods target constrained RL, where unsafe events do not terminate episodes. As acknowledged in our limitations, extending CRL to this setting remains open: its non-additive contrastive scores make Lagrangian-multiplier adjustment fundamentally difficult. (2) Modelling an explicit absorbing outcome induces false-negative issues, as explained in Section 3.2. The implementation of these two baselines faces methodological difficulties, so we leave them out for now. We hope the reviewer can understand this.

Q3 Time-limit truncation

All 3 reviewers asked about the treatment of time-limit truncation. Please see our shared response to Q1 of Reviewer 1 (8y2k). The added calibration results in Figure 6, page 7 show that the learned Z is already accurate.

As for the fraction of anchors down-weighted by truncation versus failure, unfortunately, we did not log this during training. Re-training is not practical during rebuttal. However, the top panels of Figure D.1 show low survival times early in training, particularly for humanoid. This gives some hints: we expect time-limit truncation to occur rarely during the critical early learning stage. So, excluding or re-weighting them should have little impact. We will add the complete analysis in the final paper to check this hypothesis.

Q4 Baseline retuning for modified dynamics

We did not retune Scaling-CRL; we retain its recommended hyperparameters and Safe-CRL uses the same settings. Therefore, we consider the comparison fair. In our response to Q2 of Reviewer 3 (EEAB), Safe-CRL is compared with Scaling-CRL on two Humanoid tasks in the original CRL environment. The added results (Table D.2, page 22) show that Safe-CRL retains a significant advantage over Scaling-CRL in these original environmental settings.

AuthorsOpenReview ↗

Responses to Reviewer 1LLR [part 2]

Q5 Car Goal run consistency

Both numbers are correct. Table 1 uses the final evaluation point over 5 seeds; Table 2 averages the final 10% of evaluation points over the first 3 seeds. Their identical mean survival times are a rounding coincidence.

Q6 Action-ranking reversals

This overestimation bias concerns CRL's goal-occupancy objective, not hindsight relabelling. In HER-based TD methods such as SAC-HER, overestimation arises from conventional bootstrapped TD learning rather than relabelling. They are fundamentally different.

Using an early training checkpoint on Hum. Goal and Ant Big Pitfall, we sampled replay state-goal pairs and 5 actions per pair from the Gaussian policy. We ranked these actions by ff and f+log⁡Zf+\log Z, using the critic and Z-encoder. The rankings differed in 99.8% of cases. However, judging which ranking is more reasonable is difficult in these high-dimensional action spaces (8 for Ant; 17 for Humanoid). Could you please specify what kind of visualization/relationship you would like to see? This is unclear to us for now.

Failure cases

We now add a common failure case of Safe-CRL in Figure D.6, page 22: the humanoid robot tries to avoid the pitfall pillars but loses balance and falls. Our proposed method does not guarantee failure-free behaviours, for sure.

We hope that our responses can address most of your concerns. More discussion is welcome.

Reviewer_1LLROpenReview ↗

Main Concerns Addressed, with Minor Wording Suggestions

I thank the authors for the detailed response and the additional experiments. The LSE clarification, the new Survival RL, CRL+(Z−1) and Filtered CRL comparisons, and the original spring backend results address our main concerns, and I will raise the score accordingly.

Regarding Q6, no further experiments are needed; I appreciate the added failure case.

Two small text suggestions for the final version:

The abstract’s wording (“theoretically rigorous,” “establish survival-mass correction as an effective completion of CRL”) could be aligned with the more measured conclusion, since the guarantee relies on the approximate LSE calibration and the censoring approximations in A.5. Sec. 4 still says the Humanoid actuator gear “is weakened to make the locomotion task harder,” while Appendix B now states these are the default mjx settings.

Dear Reviewer,

Thank you for your positive feedback and suggested revisions. We agree that the current wording and some expressions are not rigorous enough or well aligned. We will adjust the suggested parts in the next revision after collecting further feedback from other reviewers. Thank you again for your time and effort in reviewing our work.

Best regards,

Reviewer_8y2kOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that CRL can overvalue actions that reach a goal briefly before failure. Safe-CRL changes the weights in the contrastive loss and adds a learned survival score to the policy objective. Using only failure termination labels, it helps agents survive longer and spend more time at their goals.

Critical strengths
  1. I like how it traces the issue to the way future goals are sampled after failure termination. The analysis separates the distortion caused by normalizing each trajectory from the missing survival mass, and explains why the two proposed changes address different parts of the problem.

  2. Safe-CRL improves mean survival time and time at goal across all twelve tasks, using matched actor and critic architectures and five seeds. The component and depth ablations are useful, and the reported 2.8% training-time overhead makes the improvement practical.

Critical weaknesses

Figure 6 only shows that predicted survival mass and survival time increase together. Also, Appendix A.5 explains that contrastive weights use the observed future, while the survival estimator treats unobserved horizons after truncation as survived. The paper is open about this approximation, but it leaves unclear how accurately the implemented score recovers the quantity derived in the theory.

Questions for authors
  1. Could you compare predicted survival mass and corrected goal values with Monte Carlo estimates, particularly near time-limit truncation?

  2. On the Ant tasks with lower goal coverage, are the missed goals mainly farther away, riskier to reach, or less explored?

2 comments
AuthorsOpenReview ↗

Responses to Reviewer 8y2k

Thank you for the careful evaluation and constructive feedback. Our responses to the raised concerns are listed below. Revisions in the updated manuscript are in blue text.

Q1 Survival and goal-value estimation accuracy

First, Safe-CRL learns a contrastive score rather than an explicit goal-conditioned Q-value (Section 2.2), making direct comparison between predicted Q and Monte Carlo values impractical. Since CRL avoids TD learning, relabelling goals up to the episode time limit is a reasonable approximation. For Z-encoder learning, the 1000-step time limit is a hard boundary. Treating this truncation as failure cannot improve survival beyond that boundary anyway, so labelling it as survival is a 'neutral' choice. Truncation is also rare early in training, particularly on humanoid tasks with low initial survival times (Figure D.1, top panels), making its impact negligible.

We added an experiment to compare predicted Z with Monte Carlo estimates from saved stochastic policy checkpoints and estimates from deterministic evaluation policies across all 12 environments. The results are presented in the updated Figure 6. Predicted Z closely matches the mean Monte Carlo estimate and follows the evaluation trend. Its lower value relative to evaluation Z reflects the Gaussian training policy versus the deterministic evaluation policy (note that the Z-encoder learns the survival mass of the stochastic training policy because Safe-CRL uses the entropy maximization term to encourage exploration). This result supports the accuracy of the estimator. Please check the "Z-encoder prediction is accurate with little overhead" section on page 7 of the revised manuscript.

To examine the accuracy near truncation, we extended the maximum episode length to 1800 steps for saved final policies on Ant Goal and Ant Big Pitfall, then compared predicted Z with the mean survival mass from stochastic rollouts.

  • On Ant Big Pitfall, the MAE is 0.017 for steps 0-900 and 0.015 for steps 900-1000;
  • On Ant Goal, both round to 0.008.

These results suggest a marginal truncation effect. We cannot perform this analysis on humanoid tasks because very few episodes reach 1000 steps (top panel of Figure D.1). We have not yet included this near-truncation analysis in the revised manuscript because it might be trivial. Suggestions from the reviewer are welcome.

Q2 Causes of lower goal coverage on Ant

To be honest, we do not know why Safe-CRL has lower goal coverage than Scaling-CRL mainly on Ant tasks. Visual inspection shows that Scaling-CRL's advantage is concentrated on those farther goals, like the red spots in Figure D.5. We can also say they are less explored because the agent learns how to reach nearer goals first, then gradually explores farther goals. However, why this coverage drop is not observed for other robots remains unknown.

A possible explanation is conflicting objectives: Safe-CRL maximizes goal occupancy (Proposition 1), which need not align with goal coverage maximization. These two objectives may conflict more for Ant. We admit that we lack direct evidence for this hypothesis. We acknowledge the issue in the revision on page 6, Section 5:

  • "Table 1 shows that goal coverage degrades for Ant tasks only. Maximizing goal coverage is essentially a different objective from maximizing goal occupancy. Therefore, a possible explanation is that the two objectives conflict more for Ant. However, this hypothesis needs more investigation."

We hope that our responses can address your concerns.

AuthorsOpenReview ↗

Remind of reviewer-author discussion

Dear reviewer,

The discussion period is coming to an end soon. We would appreciate it if you could kindly share your thoughts and feedback on our responses. We will be glad to provide any further clarifications if needed. Thank you.

Best regards, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.