- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
Survival reinforcement learning (SRL) is a method for goal-conditioned RL. In essence, SRL learns to predict the probability that the goal has not yet been reached by step , and the policy is trained to minimize the discounted sum of these probabilities, i.e., the expected discounted waiting time until the goal is reached. To encourage the agent to stay in the goal region, SRL counts staying there for steps rather than a single visit as success. This paper builds on SRL with two main contributions: (1) geometric losses, which train the distance between state–action and goal embeddings to match goal-reaching times and act as a regularizer that shapes the shared embedding space; and (2) an additional occupancy head that explicitly predicts the probability that the agent is in the goal region at each future step.
Critical strengths
- The results look promising. Across the seven JaxGCRL benchmarks and all network depths, ChronoSRL learns substantially faster than CRL and AC-CRL.
- The paper is transparent about its limitations and negative results. The authors report that the survival critics fail on the arm manipulation tasks where the contrastive methods succeed. They also report that adding the geometric losses to SRL breaks learning in three of four Humanoid seeds, and that the internal ablation shows only small effects of the geometric losses (Table 3).
Critical weaknesses
- The role of the geometric losses is unclear. The learned distance never enters the value function or the policy objective. It affects the critic only indirectly, through the shared encoders. In effect it is an auxiliary loss with an interpretable target. That is a reasonable design choice, but the ablations must show that this auxiliary signal improves control. However, the internal ablation does not show this clearly (Table 3). Removing geometric losses or the permuted-times variant performs almost the same as the full method, which suggests whatever benefit the losses provide is unlikely to come from the temporal content of the targets. The representation analysis shows that the intended geometry is learned, but the paper does not show that this geometry translates into better control. The losses do help when added to the original SRL, but three of four seeds then fail to learn on Humanoid. This suggests that the losses mainly help a weak critic and can be unstable.
- The occupancy head's effect is narrow. ChronoSRL adds an occupancy head to reward staying at the goal, but it still keeps SRL’s dwell requirement of k consecutive steps. The paper does not explain why both mechanisms are needed. The ablation (k = 1, with the occupancy head) is missing, so it is unclear whether the occupancy head can replace the dwell objective or only complements it. And the occupancy head matters only on Humanoid. It seems that the occupancy value contributes to the policy objective essentially only when the agent falls.
- Several configuration changes are not ablated and may account for a large part of the gain. Besides geometric losses and occupancy head, Table 1 lists more differences from SRL. None of these is ablated individually. For example, for Ant Hardest Maze task, the results are: SRL (143), ChronoSRL with one-step actions (486), full ChronoSRL(682), w/o geometric losses (647), w/o occupancy head (681). It seems that improvement comes from these unablated settings.
Questions for authors
- Could the author clearly explain the role of the geometric losses according to the argument of Weakness 1?
- Regarding the occupancy head, could the author provide an ablation study (k = 1, with the occupancy head)? Could the author explain for different scenarios (eg., the agent falling, drifting away or overshooting with too much momentum), how or whether the occupancy head helps?
- Could the author provide more ablation studies about the modified parts compared to SRL? Especially, I am interested in these two settings: (1). ChronoSRL without geometric losses, without occupancy head; (2). SRL only with geometric losses and occupancy head.
2 comments
Thanks for the rebuttal comments and additional experiments. My questions have been answered:
- The geometric losses work as an extra (auxiliary) loss. They help the agent learn faster early in training (Table 5), but they do not change the final performance much. I think this is a fair design choice and a real contribution. The authors also give a convincing reason why the geometric losses fail on Humanoid.
- The authors explain how the occupancy head and the k-step dwell rule work together. From the study of how the humanoid fails and the ablations in Table 3, I think the occupancy head mainly helps by keeping the robot from falling.
I thank the authors for their clear answers and honest experiments, and I will raise my soundness score. However, these answers also make the contribution of the paper smaller. As I said above, the title, abstract, and Figure 1 present the geometric losses as the main idea, but they mainly act as an extra loss that speeds up early learning (about 12% higher early AUC, as a median). They have little effect on final performance. The occupancy head clearly works, but mainly in tasks where the agent can fall. In the ant mazes, most of the gain over SRL comes from action chunks and other setting changes. I suggest that the authors change the abstract and introduction so that their claims match these results. Also, the two parts are mostly independent of each other. So the paper reads more like two separate, small improvements to SRL than one connected paper built around temporal geometry.
To be honest, I find it hard to decide on my score. I see this as a borderline paper, and I will keep my score at 2, although I do not think my exact score matters much. I no longer have concerns about the technical soundness, and the experiments are careful and honest. But I think the method’s contribution is either just below the bar for ICLR or only barely above it. If the other reviewers and the AC think the contribution is enough, I am fine with that and will not argue against accepting the paper.
Thanks a lot for the careful review!
You asked us to clearly explain the role of the geometric losses. You are right that they act as an auxiliary loss with an interpretable target, since they shape the embeddings from which both heads predict the value. Their effect on control is faster learning. They speed up early learning in nearly every ant-maze setting (new Table 5), and in Ant U4-Maze, where the fastest route is not the straight line, the variants whose distance follows the time learn fastest (new Table 8). The permuted variant is one of these fast variants, because it only shuffles the targets of the time loss and its distance still follows the time. The losses also do not only help a weak critic, as they still raise the AUC of SRL with 16 layers, where SRL is much stronger on its own (new Table 6). The Humanoid failure of SRL with our losses comes from SRL's layer-normalized embeddings, and with ChronoSRL's embeddings, it learns with all four seeds (Appendix F.1).
We ran the k = 1 ablation you asked for (Appendix F.2). On Humanoid, the occupancy head alone and the dwell sequences alone each keep the humanoid at the goal for about 50 to 60% of the episode, and both together for 83%. With neither, the humanoid falls right after reaching the goal, so the two mechanisms complement each other. We also rolled out the variant without the head for your scenarios. It never overshoots and almost never drifts away, but after standing at the goal for about 400 steps, it falls in 75% of the episodes in which it reaches the goal, against 23% for ChronoSRL. This also corrects a sentence of our submission, which said that the humanoid overshoots. You are right that the occupancy head matters most when the agent can fall, and this is by design. The quadruped can fall as well, and without the head, its final time at the goal drops from 893 to 55 for velocity tracking (new column in Table 4).
We ran both settings you were interested in (new rows in Table 3) and also added all other settings to SRL one group at a time (new Table 7). In the ant mazes, these settings bring most of the gain over SRL, and with action chunks alone, the survival critic of SRL already learns faster than AC-CRL, which also uses chunks. On Humanoid, the settings help little, and the occupancy head and the geometric losses do most of the work. Adding both components to SRL in its own configuration raises its final time at the goal on Humanoid from 49% to 79% of the episode.
We hope that this answers your questions, and we would be very happy if you would consider raising your score. Thanks again for the detailed suggestions, which really improved the paper!