acceptodds
Under review as a conference paper at ICLR 2027

Survival Reinforcement Learning: Toward Scalable Self-Supervised RL

Abstract

While self-supervised Contrastive Reinforcement Learning (CRL) has shown remarkable depth-scaling capabilities, successfully using networks over 64 layers, scaled CRL still struggles with long-horizon goal-conditioned planning due to the uniformity-tolerance dilemma inherent in contrastive losses. We introduce Survival Reinforcement Learning (SRL), an online classification-based alternative that extends the survival value learning framework by maximizing the agent's dwell time at target goals. SRL bypasses the structural constraints of CRL and mitigates the "bang-bang" control solutions inherent to survival frameworks, which often induce undesirable behavior in complex dynamical systems. Evaluated across diverse robotic benchmarks, scaled SRL matches state-of-the-art CRL on manipulation tasks and outperforms it by 2x to 8x on stable, long-horizon locomotion tasks. Our results provide strong additional evidence that classification-based methods may serve as a key primitive in the broader effort to scale reinforcement learning.

Primary area:
reinforcement learning
Keywords:
Goal-conditioned reinforcement learning, Survival Analysis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_L1WeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Survival Reinforcement Learning (SRL), an online, self-supervised goal-conditioned RL algorithm that builds on Survival Value Learning (SVL). SVL reframes value estimation as predicting the distribution of first-hitting times to the goal, trained by maximum likelihood over observed and right-censored trajectories. The authors argue that Contrastive RL (CRL) is limited on long-horizon tasks by the uniformity–tolerance dilemma of InfoNCE losses, while SVL, by optimizing only the first hitting time, induces bang-bang control, i.e., agents reach the goal at full speed and overshoot or fall. To address the latter, SRL redefines the survival event as remaining within the goal region for k consecutive steps, a “dwell time at goal” formulation that requires no new loss or architecture. The authors provide a lower bound linking the k-step hitting time to goal occupancy. Empirically, SRL at depth 32 is reported to match scaled CRL with depth 64 on manipulation tasks and to substantially outperform it on AntMaze navigation.

Critical strengths
  1. The paper offers a simple, well-motivated fix that works.
  2. It is clearly written and easy to follow.
  3. It shows strong empirical results at scale.
Critical weaknesses
  1. A central motivation, the uniformity–tolerance problem of scaled CRL, is asserted but never tested.
  2. The only algorithmic change relative to SVL is the event definition - the critic is conditioned on a k-length goal sequence, and the actor is scored on (g, …, g). With such a small change, the contribution rests on the empirical comparison and on attributing its gains. Both have gaps (see Q6–Q7).
Questions for authors
  1. The critic is trained mostly on moving goal sequences (ϕ(su),…,ϕ(su+k−1))(\phi(s_u), \dots, \phi(s_{u+k-1})) but queried with constant ones (g,…,g)(g, \dots, g). How often do constant sequences appear in training, and how accurate is the critic on them?

  2. Fig. 6 shows a goal encoder that takes a single gg. How is a kk-goal sequence, with kk up to 32, fed into it?

  3. Survival targets come from observed outcomes without bootstrapping, so they cannot stitch sub-trajectories together. How does SRL handle horizons longer than the trajectories it typically observes?

  4. On the uniformity–tolerance motivation: is there any measurement of representation geometry for CRL or SRL showing that this is a real problem? Examples would be embedding distance versus temporal distance, uniformity and alignment metrics, or effective rank. SRL has comparable 128-d encoders, so the comparison seems feasible, yet the explanation appears only in the Introduction and Discussion. I raise this because the dilemma may not transfer to CRL. Wang & Liu (2021) analyze normalized embeddings with a fixed temperature τ\tau, whereas CRL's critic is an unnormalized inner product, whose effective inverse temperature is the product of the two embedding norms and is therefore learned per sample. The effect could also cut both ways. Batch negatives that are actually near-future states would hurt long horizons, but states that are close in space and far in time, such as those on opposite sides of a maze wall, may benefit from uniformity. I'd like to see both directions discussed.

  5. The motivation seems to shift from SVL. SVL was motivated by TD-bootstrapping instability, while here survival learning is motivated by InfoNCE's geometry. Could the authors clarify?

  6. The dwell-time fix seems to recover something CRL already has. CRL's occupancy target rewards staying at the goal over the whole horizon, which is stronger than requiring kk consecutive steps. The kk ablation therefore explains SRL versus SVL, not SRL versus CRL. What drives the gain over CRL?

  7. Depth scaling may not need dwell time: with k=1k = 1, success rate already scales with depth in Fig. 2. Should the scaling be credited to the survival objective and architecture, or to the dwell-time extension?

  8. The root cause of bang-bang behavior appears to be the absorbing/termination assumption rather than the reward. Without termination, the return under Eq. 1 is already discounted goal occupancy up to a constant:

    −∑γt 1out=−11−γ+∑γt 1in,-\sum \gamma^t \, \mathbb{1}^{\mathrm{out}} = -\frac{1}{1-\gamma} + \sum \gamma^t \, \mathbb{1}^{\mathrm{in}},

    where 1in\mathbb{1}^{\mathrm{in}} and 1out\mathbb{1}^{\mathrm{out}} indicate whether the next state lies inside or outside the goal region. Why keep termination and patch it with kk, rather than drop it?

2 comments

(W1 and Q4)

We agree. The abstract no longer mentions this explanation and the introduction presents it as an untested possibility. We have no measurement of representation geometry and both observations of the reviewer apply. The critic of scaled CRL is unnormalized (a negative L2 distance between the embeddings in the implementation of Wang et al.), so the analysis of Wang & Liu (2021) does not transfer directly, and uniformity can help for states that are close in space and far in time. A like-for-like measurement is also hard, because the SRL critic fuses the two embeddings in a joint head and has no distance between them.

(W2)

We agree on the scope (common response, point 1). Relative to SVL, we also move from the offline setting with a hierarchical advantage-weighted actor to the online setting with a SAC-style actor and deep residual networks. The attribution gaps are addressed in Q6 and Q7.

(Q1)

Constant sequences make up 10% of the training samples, each labelled with the k-step dwell event read from the trajectory. The other samples are k consecutive achieved goals of the same trajectory, starting at a future time (85%, sampled with weights γ^Δt) or at the current state (5%). Sec. 4.1 now states this (previously only Table 1).

(Q2)

The k goals are concatenated into one vector of dimension k · dim(g), the input of the goal encoder. Fig. 6 and its caption are corrected.

(Q3)

The reviewer is right, the critic is trained on observed outcomes without explicitly stitching sub-trajectories. Generalisation beyond observed horizons can only come from the function approximator, as for CRL, which also trains without bootstrapping. In the online setting the observed horizons grow with the policy, as every 1000-step episode is relabelled with all the goals reached along the way. The limitations now list the absence of stitching.

(Q5)

As in SVL, the motivation is a likelihood-based critic trained on the event that defines success, without bootstrapping or negative samples. In the online, large-depth setting the strongest baseline, CRL, is bootstrapping-free, so the question is how a survival critic compares with a contrastive one. The InfoNCE geometry argument was our attempt to explain the empirical gap and is toned down (W1).

(Q6)

We agree with both statements. The k ablation explains the difference to SVL, and the gain over CRL appears on the Ant mazes, where k=1, so it does not involve the k-step event.

We attribute this gain to the survival formulation in the revised discussion, but our comparison does not isolate it. The two methods differ in the objective (hitting-time likelihood vs InfoNCE on the discounted occupancy), in the critic parameterization (a joint head over the two embeddings vs a distance between them) and in the training pairs (SRL: 85% future, 5% current, 10% random states; CRL: one future state sampled with weights γ^Δt). CRL with γ=0.999 is worse than with γ=0.99 on Ant U4 (554 to 114 at depth 64) and on the enlarged Big Maze (depth 16: 158, 20 and 26 against 488, 189 and 69 at ×2, ×4 and ×8). SRL with γ=0.99 reaches 637 on U4 and 566 on Big Maze (γ=0.999: 832 and 777), against 567 and 672 for our CRL with γ=0.99, so at equal discount SRL is ahead on U4 (by 70, within the seed spread of CRL) and behind on Big Maze (by 106).

(Q7)

We agree. With k=1, the success rate already scales with depth (Fig. 2), and so does goal occupancy on the Ant mazes (U4: 488 at depth 4 and 832 at depth 32; Hardest: 195 and 768). The k-step event raises goal occupancy on Humanoid and Arm Push Easy. On Arm Push Easy it is close to its maximum at depth 4 (821 against 849 at depth 32), and on Humanoid it rises from 120 at depth 4 to about 450 at depth 8 and stays between 420 and 530 up to depth 64.

(Q8)

We agree and the text traces the bang-bang behaviour to the first-hitting objective, which ignores what happens after arrival (Eq. 4). Without termination, the return under Eq. 1 is the discounted goal occupancy up to a constant, as the reviewer writes, which is the return that the dense-reward baselines optimize (common response, point 4). The k-step event lies in between, delaying the event when the agent leaves the goal within the first k steps, but it remains a surrogate of the occupancy return and ignores the behaviour after the window. Remark 1 and Fig. 5b relate the occupancy bound to the measured occupancy.

We keep a terminating event because the survival critic needs the return to be a function of a single event time, which allows training by censored maximum likelihood with one label per sample. Without termination the return depends on the whole in/out sequence, and estimating it without bootstrapping needs a different estimator, such as the density-ratio estimator of CRL. A likelihood-based estimator of the non-terminating return would need a model of recurrent events, left for future work.

I thank the authors for the detailed rebuttal and maintain my positive score.

Reviewer_osybOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SRL extends survival learning (Tiofack et al., 2026), where a hazard model estimates the first-hitting time to a goal and the Q-function is recovered from the survival probability (Eq. 4). The paper changes the event from first entry into the goal region to k consecutive steps inside it (Eq. 8), to remove the bang-bang incentive of the first-hitting-time objective. The hazard critic is trained online with hindsight relabeling and a SAC-style actor on the residual architecture of Wang et al. (2025). The evaluation compares SRL with scaled CRL on 12 JaxGCRL tasks and on 2×/4× enlarged Ant mazes, with a depth sweep, an ablation on k, and a goal-occupancy lower bound (Prop. 1).

Critical strengths

The problem is well identified.

  • Fig. 2: with k=1k=1, success rate reaches ≈100%\approx 100\% on Humanoid and Arm Push Easy while goal occupancy stays near zero, i.e., success rate hides the failure to stay at the goal.
  • The fix is a single change to the event definition, with no change to loss or architecture.
Critical weaknesses

W1. The claims go beyond what the experiments establish.

  • The critic, loss, architecture, environments, metric and most baseline numbers are inherited from Tiofack et al. (2026) and Wang et al. (2025); the new element is the kk-step event.
  • The per-environment kk in Fig. 1 is not reported. Sec. 5.2 and the Limitations suggest k=1k=1 on the Ant mazes, the six environments behind the "2x–8x" claim; the paper's own modification then appears to matter only on Humanoid and Arm Push Easy, where SRL does not exceed CRL.
  • The scalability claim in the title is not directly supported:
    • the paper does not identify which of its additions would affect depth scaling; the properties cited in Sec. 6 belong to the inherited objective and architecture;
    • Fig. 1 compares SRL at depth 32 with CRL at depth 64 and does not compare their responses to increasing depth under a common setup; the only depth sweep (Fig. 7) has no CRL curves;
    • SRL at depth 4 already beats CRL at depth 64 on Ant U4/U5, so the gains cannot be attributed to depth.
  • The results support strong performance on the evaluated tasks more directly than a comparative scaling claim.

W2. The manuscript is not self-contained enough to reproduce the method or verify the results.

  • The derivation of Eq. 4, the hazard-head parameterization and the likelihood actually optimized are deferred to Tiofack et al. (L185, L712, L730).
  • Eqs. 7 and 10 condition on a goal sequence g(k)g^{(k)}, but the critic takes a single goal (Fig. 6, L702) and the actor queries (g,…,g)(g,\dots,g). Which sequences the critic is trained on is not stated; matching a changing sequence and remaining at one target are different events.
  • Algorithm 1 (main text) omits the hindsight relabeling that Algorithm 2 (appendix) relies on; neither says how g′g' is sampled or how Table 1's sampling mixture relates to it.
  • The environments are not described, and no numeric table accompanies Fig. 1, so the abstract's figures cannot be checked.
Questions for authors

Clarification requests

  1. Which kk is used per environment in Fig. 1? What are the Ant maze results with k>1k>1?
  2. Can you show CRL and SRL at matched depths (4–64) with the same seeds and band statistic, plus parameter counts?
  3. Is the critic trained only on constant sequences (g′,…,g′)(g',\dots,g')? If so, why is Eq. 7 stated in general form; if not, how is the sequence encoded? Please also report the censoring fraction vs. kk.
  4. For the CRL curves taken from Wang et al.: are γ\gamma, environment steps, seeds, ε\varepsilon, evaluation rollouts and the band statistic identical to SRL's? For the CRL runs in Fig. 4, which γ\gamma and goal-sampling scheme were used? With γ=0.99\gamma=0.99, geometric future-state sampling rarely reaches goals 400+ steps away, which alone could explain the collapse on the 4×4\times maze.
  5. Is termination on goal arrival disabled? How is this reconciled with the absorbing-goal assumption behind Eqs. 4 and 9?

Suggestions

  1. A dense-dwell baseline (SAC+HER with a per-step in-goal reward) to separate the survival formulation from rewarding dwell time.
  2. Readability and reporting:
    • The main text ends a page below the limit while Eqs. 11–12 duplicate Eq. 6 and Algorithms 1–2 duplicate each other with different notation; consider moving Algorithm 2 to the main text.
    • Numeric tables (mean ± CI, seeds) and an environment table.
    • Row labels on Fig. 9; the kk/KK symbol clash in Eq. 15.
    • The duplicated Bortkiewicz 2024/2025 reference; "2x–8x" (abstract) vs. "two to four times" (Sec. 5.3).
4 comments

(W1) Claims and experiments:

The reviewer's reading is correct, we use k=1 on all six Ant mazes. The k-step event is active on the arm and humanoid tasks, and the k of every environment is now reported (Q1).

We agree that the results support strong performance more directly than a comparative scaling claim and we have adapted the title and abstract. However, depth does help SRL on the harder mazes (Hardest: 195 at depth 4 and 768 at depth 32; Big Maze: 357 and 777) and we see the properties cited in Sec. 6 (objective and architecture) as the main drivers.

(W2) Self-containedness:

The revision adds the following

  • The derivation of Eq. 4 in Sec. 3, and Appendix A.3 on the hindsight labels
  • A paragraph in Sec. 4.1 on how goal sequences are sampled and encoded (Q3)
  • The full algorithm, with hindsight relabeling, in the main text (Algorithm 1)
  • The per-environment hyperparameters (Table 2) and numeric tables with mean ± std for Figs. 1, 7 and 4 (Tables 3, 4 and 5)
  • The definition of depth. An SRL critic of depth D has two encoders of depth D/2 and a head trunk of depth D, which gives 2D block layers, as the two depth-D encoders of CRL. The submission stated depth D for all three networks, which is corrected.

(Q1)

We use k=1 on the six Ant mazes, k=20 on the three arm tasks (Arm Push Easy, Arm Push Hard, Arm Binpick Hard) and k=32 on the three humanoid tasks (Humanoid, Humanoid U-Maze, Humanoid Big Maze). On the Ant mazes, larger k slightly lowers goal occupancy. On U4 at depth 32 it is 832 for k=1 and 745 for k=20, which is still above CRL (567).

(Q2)

Table 4 reports both methods at depths 4, 8, 16, 32 and 64 as mean ± std. The per-depth CRL values are those released by Wang et al. (5 seeds), complemented by our own CRL runs (4 seeds), so the seeds cannot be matched. Rerunning CRL at every depth only for this is too costly. At equal nominal depth the SRL critic has 7% more parameters at depth 32 and 3% more at depth 64, so SRL at depth 32 has about half the parameters of CRL at depth 64 (4.6M and 8.5M on Ant).

(Q3)

The critic is trained on general sequences. With probability 0.85 the goal sequence is the k consecutive achieved goals from a future time of the same trajectory (sampled with weights γ^Δt), with probability 0.05 the k achieved goals from the current state, and with probability 0.10 a constant sequence (g′, …, g′) built from a random state of the sampled trajectory slice. The new paragraph "Training sequences" in Sec. 4.1 states this (previously it was only in Table 1).

Between 2 and 6% of the samples are censored for every k from 1 to 32 on Ant, Arm Push Easy and Humanoid. The future and current sequences (90%) are events by construction, so only random sequences can be censored. A sample without a later hit is labelled censored at the fixed horizon H = 1000, whatever the number of remaining steps (Appendix A.3).

(Q4)

They are not identical in every aspect. With the original code of Wang et al. at depth 64, CRL with γ=0.999 is worse than with γ=0.99, with goal occupancy dropping from 554 to 114 on U4 and from 662 to 428 on Arm Push Easy.

For Fig. 4, CRL was trained with our implementation, γ=0.99 and geometric sampling of future states as in Wang et al., and SRL uses γ=0.999 with the same sampling. The explanation proposed by the reviewer is plausible. In CRL, γ sets both the horizon of the objective and the distribution of positives, and raising it to 0.999 degrades training on the standard maze. We have since run CRL with γ=0.999 on the enlarged mazes (depth 16, otherwise the same settings as the γ=0.99 runs). It reaches 158, 20 and 26 on ×2, ×4 and ×8, below the γ=0.99 runs (488, 189 and 69) and well below SRL with γ=0.999 (777, 499 and 137, Table 5), so raising the discount of CRL does not close the gap.

(Q5)

Termination on arrival is disabled in the environment and Sec. 5.1 now says so. Episodes last 1000 steps and end early only when the robot becomes unhealthy (locomotion tasks). The absorbing goal defines the objective of Eq. 4. The training label is the first hitting time read from the logged trajectory, and it does not depend on what the agent does after arrival, so the hitting-time distribution of the terminating problem can be estimated from non-terminating data. The k=1 objective leaves the behaviour after arrival unconstrained, which is the failure shown in Fig. 2.

(S1)

This baseline already exists in the benchmark: SAC+HER in JaxGCRL receives a reward of 1 at every step inside the goal region (a dense dwell reward) and does not terminate on arrival. Wang et al.'s results for it are in the common response (point 4). We did not run a further dense-dwell variant.

(S2)

All points are adopted: Algorithm 2 moved to the main text, numeric tables and per-environment settings added, row labels in Fig. 9, the k/K clash in Eq. 15 resolved, the duplicated reference removed and the inconsistent factors removed.

Thank you for the thorough response and revision. I appreciate the additional CRL runs, the new tables, and the openness in reporting results that do not favour SRL.

Most of my concerns about presentation and overclaiming are addressed (W2, Q1, Q4, Q5, S2, and the title).

My main concern remains the contribution. The k-step event works as intended, but among the six tasks where it is active, SRL is clearly ahead of CRL only on Humanoid U-Maze (at depth 32). The main gain over CRL is on the Ant mazes, where k=1, and that comparison does not isolate where the gain comes from (the discount differs, and the CRL numbers come from two sources). I therefore keep my rating.

Only a few days remain in the discussion, and I would like you to be able to use them well, so let me be transparent: addressing the comments below will not change my score. They are suggestions for the paper and for your follow-up work.

  1. Baseline in Fig. 1. Fig. 1 plots the released CRL curves (286 on U4 and 441 on Big Maze, depth 64), while your own depth-32 CRL runs reach 567 and 672. Please explain the gap in the paper and use one baseline consistently.
  2. Equal discount. At γ=0.99, your responses report 637 (U4) and 566 (Big Maze) for SRL against 567 and 672 for CRL. Please add this to Table 3. If the claim is that SRL remains trainable at γ=0.999 where CRL degrades, it would be better to say so directly.
  3. Scaling statements. In Table 4, SRL saturates by depth 8–16 on five of the six Ant mazes; on Hardest it peaks at depth 32 and drops to 294 ± 230 at depth 64, the only Ant maze run at that depth. Please remove the scaling claims that remain in the Discussion and in the last sentence of the abstract, or limit them to what Table 4 shows. The abstract also says SRL matches CRL on manipulation, which Table 3 does not support for Arm Push Hard and Arm Binpick Hard.

Minor

  • Algorithm 1 notation: c is both the rollout horizon and the censoring time, and τt\tau_t (line 10) and tht_h (line 12) are used for the same quantity.
  • A.3 censors at H=1000 regardless of the remaining steps.
  • Eq. 13 samples (s, g) without saying how g is derived from the stored sequences.
  • Tables 1 and 2 disagree on batch size, and K=200 on the humanoid mazes does not fit H=1000.

If I have misread any of the results above, please do correct me.

Thank you for the second reading and for stating clearly what affects your rating. You have not misread the results and the comment below points at a mistake on our side, which we have addressed now.

Main concern (where the gain on the Ant mazes comes from)

We ran a controlled comparison in one setting. Within each maze row, the four columns differ only in the method and the discount. All cells use the same codebase, depth 32, batch size 512 and the evaluation protocol, on the training layout with all goals of the maze (see comment 1 below). Values are goal occupancy out of 1000 at 100M/200M environment steps, mean ± std over 4 seeds of the last 5 evaluations.

SRL γ=0.99SRL γ=0.999CRL γ=0.99CRL γ=0.999
Ant U4637 ± 7824 ± 15574 ± 52184 ± 57
Ant U5 (200M)610 ± 36787 ± 17395 ± 65119 ± 51
Ant Big Maze566 ± 26787 ± 30640 ± 13116 ± 65
  • The best value in every row is SRL with γ=0.999 against CRL at its better discount γ=0.99
  • With γ=0.99, SRL scores 637, 610 and 566. It is then still ahead of CRL on U4 (by 63) and on U5 (by 215) and behind on Big Maze (by 74)
  • SRL trains at both discounts, whereas CRL collapses at γ=0.999. Between the two discounts, SRL loses 23-28% of its occupancy and CRL 68-82%

The two methods use the discount differently. In SRL, the critic is trained on the hitting time of the goal, a label that does not involve γ (Eqs. 10 and 11). γ enters in the sampling weights of the hindsight goals and when the value is computed from the survival curve (Eq. 9). In CRL, the positives of the contrastive loss are drawn from the γ-discounted future-state distribution, so γ defines the quantity that the critic estimates. This is our reading of the result.

As you propose in comment 2, the paper will state directly that SRL benefits from γ=0.999, where CRL degrades, and that its gain over CRL on U4 and Big Maze comes from this.

Comment on the CRL baseline in Fig. 1 We sincerely thank the reviewer for pointing out this gap. It comes from a mismatch we were not aware of until now. Following the JaxGCRL paper (Bortkiewicz et al., Sec A.5, page 19) all our runs used identical environments for training and evaluation. However, the eval_env_id field in the JSON file of CRL curves released by Wang et al (https://github.com/wang-kevin3290/scaling-crl/blob/main/results_data/fig1_main.json) shows that they evaluated Ant U4, Ant U5 and Ant Big Maze on different layouts that keep only distant goals: 4 of 10 goal cells on U4, 8 of 16 on U5, 9 of 25 on Big Maze. This affects only these three environments. To answer Q4 in our first response above, running the original code of Wang et al. and evaluating on all goals, our depth-64 runs reach 554 on U4 against 286 in their curve.

Accordingly, in the new revised PDF we updated Fig. 1 and Table 3 to use the CRL runs of the table above on these three mazes, so that every curve is trained and evaluated on the same layout, as described in the JaxGCRL paper (App. A.5). For all other tasks, Wang et al. already used the same layout for training and evaluation. So, in our revised PDF all runs (both SRL and CRL) on all environments are now trained and evaluated on the same setup. We thank the reviewer again for this question, which led to a critical fix in our paper, and we apologize for not catching this sooner.

Thank you also for comments 2 and 3 and for the minor points which we will integrate into the final version.

The attribution of the Ant results was the main open question. If the table resolves it, we would be grateful if you would reconsider the rating. If anything is missing, we will try to add it before the discussion closes.

Thank you for the additional experiments. The controlled comparison is what I had hoped to see, and tracing the gap in Fig. 1 to the evaluation layouts of the released curves, then correcting the figure and Table 3, is a real improvement to the paper. I appreciate the care you have put into this discussion.

I agree that your table answers the attribution question: SRL trains at γ=0.999, where CRL degrades, and at equal discount it is ahead on U5, narrowly ahead on U4 and behind on Big Maze.

Attribution was one part of my concern. The concern itself, as in my previous comment and the first point of W1, is the size of the contribution beyond SVL. In my reading:

  • The k-step event addresses a limitation of the first-hitting objective that CRL does not have. On the six tasks where it is active, SRL is clearly ahead of CRL on one (at depth 32) and behind on three.
  • The gain over CRL is real: 1.2–2× on the standard Ant mazes with the evaluation matched, and larger on the enlarged maze. But k=1 there, so it comes from the survival critic that SRL takes from SVL, mostly through its tolerance of γ=0.999.

The revision rightly dropped the scaling claim from the title, and the corrected evaluation reduced the Ant gain from the original 2x–8x. So the contribution the evidence supports is narrower than the one the submission presented: an online adaptation of SVL that tolerates γ=0.999 where CRL does not, a small and well-motivated modification, and a careful empirical comparison. This is useful, but in my assessment it is still below the bar, so I keep my rating. This is a judgement of scope, not of missing experiments.

Thank you again for your effort and openness throughout the rebuttal. The paper is clearly better for it.

Reviewer_rPkgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Survival Reinforcement Learning (SRL), an online self-supervised goal-conditioned RL algorithm. The authors identify that standard survival learning frameworks, which minimize the first-hitting-time to a goal, often lead to undesirable "bang-bang" control solutions (e.g., a humanoid falling upon reaching the target). To address this, SRL extends the survival objective by introducing a dwell-time-at-goal formulation, requiring the agent to match a sequence of k consecutive goal states. Empirically, the authors demonstrate that a 32-layer SRL outperforms the 64-layer Contrastive RL (CRL) baseline on the JaxGCRL benchmark, particularly in stable locomotion and long-horizon navigation tasks.

Critical strengths

Identifies a practical and important issue: The paper correctly points out that standard survival learning (and many goal-reaching RL methods) ignores post-goal behavior, leading to "bang-bang" control and instability in complex dynamical systems like Humanoids.

Strong empirical performance on specific tasks: The proposed method shows impressive scaling behavior and stability on AntMaze and Humanoid environments, achieving near-optimal goal occupancy.

Computational efficiency: Leveraging Jax and Brax for GPU-accelerated online GCRL experiments is a solid engineering choice that allows for extensive depth-scaling ablations.

Critical weaknesses

(1) Incremental Novelty and Overclaiming the Framework Extension. The core contribution of the paper is forcing the agent to remain in the goal region for k consecutive steps. In standard RL, this is trivially achieved by modifying the reward function (e.g., providing a dense, continuous reward for being inside the goal region) or altering the episode termination conditions. Framing this simple heuristic as a fundamental extension of the survival learning framework (Eq. 7 & 8) feels like an overclaim. The mathematical formulation of the extended hitting time Tg(k)πT^{\pi}_{g^{(k)}} is essentially just a re-indexing of the success condition over a sliding window of k steps, rather than a novel theoretical advancement in survival analysis.

Missing Crucial Ablations on the Baseline (Unfair Comparison). The paper attributes the superiority of SRL over CRL to the fact that the survival objective bypasses the uniformity-tolerance dilemma of InfoNCE losses. However, the authors fail to isolate the contribution of the survival framework from the dwell-time trick.

Trivial Theoretical Contribution. Proposition 1 (Goal-occupancy lower bound) is presented as a theoretical justification for the method. However, the proof (Appendix A.1) relies on basic properties of indicator functions and the linearity of expectation. It does not provide any deep insight into why or how the survival formulation scales better with network depth compared to contrastive methods. It merely states a mathematically obvious fact: if the probability of exiting the goal decreases, the occupancy increases.

Contradiction with the "Scalable" Title. The title claims "Toward Scalable Self-Supervised RL". Yet, as acknowledged in the limitations, SRL completely fails on Arm Push Hard due to exploration bottlenecks (collapsing into trivial HER targets). Since complex manipulation and exploration are primary drivers for scaling RL, failing on a standard manipulation benchmark weakens the overarching claim of the paper.

Questions for authors

Have you experimented with applying the k-step sequence matching to the InfoNCE loss of the CRL baseline? If CRL also benefits from this "dwell-time" trick, how do you justify that the survival framework itself is the key to scalability?

Given the stark difference in required k values between quasi-static manipulation (k=4) and dynamic balance (k=32), how would a practitioner deploy SRL on a novel, unseen robotic task without expensive hyperparameter tuning?

Does the right-censored likelihood optimization introduce any additional computational overhead or instability compared to the standard InfoNCE loss, especially when k is large?

1 comment

(W1) Incremental novelty and framing:

We agree that replacing first arrival by k consecutive in-goal steps is a small, targeted change and the revised text presents it as such (common response, point 1).

We wanted to obtain this dwell-goal behavior for the hitting-time survival critic, where we can neither densify the reward nor change the termination of the episode. A dense per-step reward gives a different learning problem. The k-step event keeps a single event time per sample, so the critic, loss and architecture stay as they are.

(W2 and Q1) Separating the survival critic from the k-step event:

The value of k separates the two. The six Ant mazes use k=1, so the comparison with CRL on these tasks isolates the effect of the survival critic. On Arm Push Easy at depth 16, goal occupancy is 21, 459, 478, 581 and 763 for k = 1, 4, 8, 16 and 32 that shows the effect of the dwell-time. In Fig. 2, the effect of k is studied in terms of success rate and goal occupancy.

The critic of CRL estimates the discounted occupancy of the goal and has no hitting time in its formulation. Every future step at the goal contributes to it and nothing terminates at arrival, so the preference for remaining at the goal, which the k-step event gives the survival critic, is already part of the CRL objective. We tested this with CRL whose positive is a sequence of k consecutive future goals of the same trajectory (the actor queries the critic with the goal repeated k times). On Humanoid with k=6, goal occupancy is 749 ± 20, against 756 ± 10 for CRL with a single goal, so the goal sequence gives no benefit at the end. On Humanoid U-Maze with k=32, occupancy ends at 63 ± 21 against 23 ± 17 for CRL, compared with 203 ± 35 for SRL (k=32).

(W3) Proposition 1:

We agree, the statement follows from elementary reformulations and says nothing about depth scaling. Its role is to relate the optimized quantity, the k-step hitting time, to the reported metric. In the revision it is a remark. We make no theoretical claim on why a survival critic would scale better than a contrastive one. The comparison is empirical, and the discussion offers only a qualitative interpretation (the critic is trained by maximum-likelihood classification of hitting times).

(W4) Title and Arm Push Hard:

The title is now "Survival Reinforcement Learning: Self-Supervised RL with Dwell-Time at Goal" (common response, point 1). On Arm Push Hard, SRL reaches a goal occupancy of 185 vs CRL 455. So SRL learns the task in part but stays well below scaled CRL. The actor is an unmodified SAC actor, chosen to study the critic in isolation. As mentioned, we attribute the result on Arm Push Hard to exploration. Bortkiewicz et al. (2025, App. A.5) describe a related failure of SAC+HER on Pusher, where hindsight relabeling produces trivial goals because the random policy rarely touches the object. Using the predicted hitting-time distribution to guide exploration is left for future work.

(Q2) Choosing k:

k is a task-dependent hyperparameter. The experiments use three values, k=1 on Ant, k=20 on the arm tasks and k=32 on the humanoid tasks. In Section 5.5 and Fig. 5, goal-occupancy on Humanoid and Arm Push Easy consistently improves as k increases, However, on the Ant mazes, larger k slightly lowers goal occupancy. On U4 at depth 32 it is 832 for k=1 and 745 for k=20, which is still above CRL (567). The dependence of k on the task is stated in the limitations.

(Q3) Overhead and stability:

The loss evaluates K+1 hazard logits per sample (K ≤ 1000), compared with the B×B similarity matrix of InfoNCE (B=512), so its cost is of the same order. Critic parameter counts at the same nominal depth differ by 7% at depth 32 and 3% at depth 64, SRL being larger (on Ant, SRL has 4.57M and CRL 4.29M parameters at depth 32, and 8.81M and 8.53M at depth 64). A larger k affects the hindsight relabeling, whose memory grows linearly with k, and the input size of the goal encoder. On one H100 at depth 32 (Humanoid, batch 512), CRL runs at 4,958 steps/s and SRL at 4,574 (k=1), 4,427 (k=20) and 4,373 (k=32), i.e. 88 to 92% of the speed of CRL.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers for their careful reading and constructive feedback. Several points concern more than one reviewer and we answer them here once. We have uploaded a revised PDF that highlights the modifications. We will keep this version until the end of the rebuttal period and then replace it with a clean version.

1. Title and claims. Reviewers rPkg and osyb find the scalability claim in the title not fully supported by the experiments and we agree. The updated title is now "Survival Reinforcement Learning: Self-Supervised RL with Dwell-Time at Goal" and the abstract and introduction no longer make such claim. Relative to SVL (Tiofack et al., 2026), the algorithmic change is a targeted redefinition of the event and the setting is online, with a SAC-style actor and deep residual networks.

The revised paper makes three points:

  • an online survival critic with the first-hitting event (k=1) and a SAC-style actor reaches a higher goal occupancy than scaled CRL on the ant mazes, at the same nominal depth and evaluation protocol (first row Fig. 1)
  • with k=1, the success rate for SRL is close to 1 on Humanoid and Arm Push Easy while the agent spends less than 3% of the episode at the goal (almost 0 goal occupancy in Fig. 2)
  • defining the event as k consecutive in-goal steps removes this gap without changing the loss or architecture

2. Value of k per environment. The submission did not report it specifically for every environment. We use k=1 on the six Ant mazes, k=20 on the three arm tasks and k=32 on the three humanoid tasks (new Table 2 in App A.4 with all per-environment settings).

3. Uniformity-tolerance less prominent. The uniformity-tolerance explanation of CRL's long-horizon performance, which we did not test, is removed from the abstract and the discussion. However, we keep it in the introduction as a possible explanation that we state we do not test. Wang & Liu (2021, arXiv:2012.09740) show for contrastive losses that pursuing uniformity of the embeddings makes the loss less tolerant to semantically similar samples. If this carries over to CRL, states that are close play the role of similar samples. We have neither derived this nor analyzed the representations, so it remains only a hypothesis that is consistent with our empirical results.

4. Dense per-step reward without termination. Reviewers rPkg, osyb and L1We ask about rewarding every step spent at the goal. This variant is part of the benchmark. The SAC, SAC+HER and TD3+HER baselines of JaxGCRL receive a reward of 1 at every step inside the goal region (a dense dwell reward) and do not terminate on arrival. Wang et al. (2025, Figs. 12–13) report that scaled CRL outperforms them on 8 of 10 tasks, the exceptions being the two humanoid mazes where SAC learns faster early on, and that they do not improve beyond depth 4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.