- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Survival Reinforcement Learning (SRL), an online, self-supervised goal-conditioned RL algorithm that builds on Survival Value Learning (SVL). SVL reframes value estimation as predicting the distribution of first-hitting times to the goal, trained by maximum likelihood over observed and right-censored trajectories. The authors argue that Contrastive RL (CRL) is limited on long-horizon tasks by the uniformity–tolerance dilemma of InfoNCE losses, while SVL, by optimizing only the first hitting time, induces bang-bang control, i.e., agents reach the goal at full speed and overshoot or fall. To address the latter, SRL redefines the survival event as remaining within the goal region for k consecutive steps, a “dwell time at goal” formulation that requires no new loss or architecture. The authors provide a lower bound linking the k-step hitting time to goal occupancy. Empirically, SRL at depth 32 is reported to match scaled CRL with depth 64 on manipulation tasks and to substantially outperform it on AntMaze navigation.
Critical strengths
- The paper offers a simple, well-motivated fix that works.
- It is clearly written and easy to follow.
- It shows strong empirical results at scale.
Critical weaknesses
- A central motivation, the uniformity–tolerance problem of scaled CRL, is asserted but never tested.
- The only algorithmic change relative to SVL is the event definition - the critic is conditioned on a k-length goal sequence, and the actor is scored on (g, …, g). With such a small change, the contribution rests on the empirical comparison and on attributing its gains. Both have gaps (see Q6–Q7).
Questions for authors
-
The critic is trained mostly on moving goal sequences but queried with constant ones . How often do constant sequences appear in training, and how accurate is the critic on them?
-
Fig. 6 shows a goal encoder that takes a single . How is a -goal sequence, with up to 32, fed into it?
-
Survival targets come from observed outcomes without bootstrapping, so they cannot stitch sub-trajectories together. How does SRL handle horizons longer than the trajectories it typically observes?
-
On the uniformity–tolerance motivation: is there any measurement of representation geometry for CRL or SRL showing that this is a real problem? Examples would be embedding distance versus temporal distance, uniformity and alignment metrics, or effective rank. SRL has comparable 128-d encoders, so the comparison seems feasible, yet the explanation appears only in the Introduction and Discussion. I raise this because the dilemma may not transfer to CRL. Wang & Liu (2021) analyze normalized embeddings with a fixed temperature , whereas CRL's critic is an unnormalized inner product, whose effective inverse temperature is the product of the two embedding norms and is therefore learned per sample. The effect could also cut both ways. Batch negatives that are actually near-future states would hurt long horizons, but states that are close in space and far in time, such as those on opposite sides of a maze wall, may benefit from uniformity. I'd like to see both directions discussed.
-
The motivation seems to shift from SVL. SVL was motivated by TD-bootstrapping instability, while here survival learning is motivated by InfoNCE's geometry. Could the authors clarify?
-
The dwell-time fix seems to recover something CRL already has. CRL's occupancy target rewards staying at the goal over the whole horizon, which is stronger than requiring consecutive steps. The ablation therefore explains SRL versus SVL, not SRL versus CRL. What drives the gain over CRL?
-
Depth scaling may not need dwell time: with , success rate already scales with depth in Fig. 2. Should the scaling be credited to the survival objective and architecture, or to the dwell-time extension?
-
The root cause of bang-bang behavior appears to be the absorbing/termination assumption rather than the reward. Without termination, the return under Eq. 1 is already discounted goal occupancy up to a constant:
where and indicate whether the next state lies inside or outside the goal region. Why keep termination and patch it with , rather than drop it?
2 comments
I thank the authors for the detailed rebuttal and maintain my positive score.
(W1 and Q4)
We agree. The abstract no longer mentions this explanation and the introduction presents it as an untested possibility. We have no measurement of representation geometry and both observations of the reviewer apply. The critic of scaled CRL is unnormalized (a negative L2 distance between the embeddings in the implementation of Wang et al.), so the analysis of Wang & Liu (2021) does not transfer directly, and uniformity can help for states that are close in space and far in time. A like-for-like measurement is also hard, because the SRL critic fuses the two embeddings in a joint head and has no distance between them.
(W2)
We agree on the scope (common response, point 1). Relative to SVL, we also move from the offline setting with a hierarchical advantage-weighted actor to the online setting with a SAC-style actor and deep residual networks. The attribution gaps are addressed in Q6 and Q7.
(Q1)
Constant sequences make up 10% of the training samples, each labelled with the k-step dwell event read from the trajectory. The other samples are k consecutive achieved goals of the same trajectory, starting at a future time (85%, sampled with weights γ^Δt) or at the current state (5%). Sec. 4.1 now states this (previously only Table 1).
(Q2)
The k goals are concatenated into one vector of dimension k · dim(g), the input of the goal encoder. Fig. 6 and its caption are corrected.
(Q3)
The reviewer is right, the critic is trained on observed outcomes without explicitly stitching sub-trajectories. Generalisation beyond observed horizons can only come from the function approximator, as for CRL, which also trains without bootstrapping. In the online setting the observed horizons grow with the policy, as every 1000-step episode is relabelled with all the goals reached along the way. The limitations now list the absence of stitching.
(Q5)
As in SVL, the motivation is a likelihood-based critic trained on the event that defines success, without bootstrapping or negative samples. In the online, large-depth setting the strongest baseline, CRL, is bootstrapping-free, so the question is how a survival critic compares with a contrastive one. The InfoNCE geometry argument was our attempt to explain the empirical gap and is toned down (W1).
(Q6)
We agree with both statements. The k ablation explains the difference to SVL, and the gain over CRL appears on the Ant mazes, where k=1, so it does not involve the k-step event.
We attribute this gain to the survival formulation in the revised discussion, but our comparison does not isolate it. The two methods differ in the objective (hitting-time likelihood vs InfoNCE on the discounted occupancy), in the critic parameterization (a joint head over the two embeddings vs a distance between them) and in the training pairs (SRL: 85% future, 5% current, 10% random states; CRL: one future state sampled with weights γ^Δt). CRL with γ=0.999 is worse than with γ=0.99 on Ant U4 (554 to 114 at depth 64) and on the enlarged Big Maze (depth 16: 158, 20 and 26 against 488, 189 and 69 at ×2, ×4 and ×8). SRL with γ=0.99 reaches 637 on U4 and 566 on Big Maze (γ=0.999: 832 and 777), against 567 and 672 for our CRL with γ=0.99, so at equal discount SRL is ahead on U4 (by 70, within the seed spread of CRL) and behind on Big Maze (by 106).
(Q7)
We agree. With k=1, the success rate already scales with depth (Fig. 2), and so does goal occupancy on the Ant mazes (U4: 488 at depth 4 and 832 at depth 32; Hardest: 195 and 768). The k-step event raises goal occupancy on Humanoid and Arm Push Easy. On Arm Push Easy it is close to its maximum at depth 4 (821 against 849 at depth 32), and on Humanoid it rises from 120 at depth 4 to about 450 at depth 8 and stays between 420 and 530 up to depth 64.
(Q8)
We agree and the text traces the bang-bang behaviour to the first-hitting objective, which ignores what happens after arrival (Eq. 4). Without termination, the return under Eq. 1 is the discounted goal occupancy up to a constant, as the reviewer writes, which is the return that the dense-reward baselines optimize (common response, point 4). The k-step event lies in between, delaying the event when the agent leaves the goal within the first k steps, but it remains a surrogate of the occupancy return and ignores the behaviour after the window. Remark 1 and Fig. 5b relate the occupancy bound to the measured occupancy.
We keep a terminating event because the survival critic needs the return to be a function of a single event time, which allows training by censored maximum likelihood with one label per sample. Without termination the return depends on the whole in/out sequence, and estimating it without bootstrapping needs a different estimator, such as the density-ratio estimator of CRL. A likelihood-based estimator of the non-terminating return would need a model of recurrent events, left for future work.