acceptodds
Under review as a conference paper at ICLR 2027

ChronoSRL: Temporal Geometry for Self-Supervised Reinforcement Learning

Abstract

A goal that is close in space can be far away in time. Obstacles, terrain, and the agent's own capabilities determine how long it takes to get there. Yet, critics in contrastive and survival reinforcement learning do not measure the distances in their representation space in units of time. We therefore introduce ChronoSRL, which gives the critic's embeddings an explicit temporal geometry. The distance between state–action and goal embeddings is trained to match the time that the agent takes to reach the goal (goal-reaching time), while goals that were not reached, and goals from other trajectories, are pushed at least one discount horizon away. Furthermore, reaching a goal quickly once does not mean that reaching it is reliable in general, so the policy should not follow the temporal distance directly. Instead, we build on survival reinforcement learning and predict from our temporal embeddings not only the full distribution of goal-reaching times but also the time spent near the goal. Thereby, the policy is trained to favor actions that reach the goal sooner and more reliably and that keep the agent near it. ChronoSRL learns faster and reaches higher performance than contrastive, action-chunked contrastive, and survival reinforcement learning baselines on seven standard locomotion and navigation benchmarks, even with much smaller networks. To test the limits of self-supervised reinforcement learning, we introduce velocity tracking, goal-position reaching, and box climbing tasks with a quadruped robot in a realistic sim-to-real locomotion setup, and show how the shaping terms that are typical for robotics can be naturally incorporated into our framework. ChronoSRL is the only one of the tested self-supervised reinforcement learning methods that learns to stay at the commanded velocities and goal positions, and climbs the highest boxes.

Primary area:
reinforcement learning
Keywords:
Self-Supervised Reinforcement Learning, Contrastive Reinforcement Learning, Goal-Conditioned Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_8QpCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

Survival reinforcement learning (SRL) is a method for goal-conditioned RL. In essence, SRL learns to predict the probability that the goal has not yet been reached by step tt, and the policy is trained to minimize the discounted sum of these probabilities, i.e., the expected discounted waiting time until the goal is reached. To encourage the agent to stay in the goal region, SRL counts staying there for kk steps rather than a single visit as success. This paper builds on SRL with two main contributions: (1) geometric losses, which train the distance between state–action and goal embeddings to match goal-reaching times and act as a regularizer that shapes the shared embedding space; and (2) an additional occupancy head that explicitly predicts the probability that the agent is in the goal region at each future step.

Critical strengths
  1. The results look promising. Across the seven JaxGCRL benchmarks and all network depths, ChronoSRL learns substantially faster than CRL and AC-CRL.
  2. The paper is transparent about its limitations and negative results. The authors report that the survival critics fail on the arm manipulation tasks where the contrastive methods succeed. They also report that adding the geometric losses to SRL breaks learning in three of four Humanoid seeds, and that the internal ablation shows only small effects of the geometric losses (Table 3).
Critical weaknesses
  1. The role of the geometric losses is unclear. The learned distance dθd_{\theta} never enters the value function or the policy objective. It affects the critic only indirectly, through the shared encoders. In effect it is an auxiliary loss with an interpretable target. That is a reasonable design choice, but the ablations must show that this auxiliary signal improves control. However, the internal ablation does not show this clearly (Table 3). Removing geometric losses or the permuted-times variant performs almost the same as the full method, which suggests whatever benefit the losses provide is unlikely to come from the temporal content of the targets. The representation analysis shows that the intended geometry is learned, but the paper does not show that this geometry translates into better control. The losses do help when added to the original SRL, but three of four seeds then fail to learn on Humanoid. This suggests that the losses mainly help a weak critic and can be unstable.
  2. The occupancy head's effect is narrow. ChronoSRL adds an occupancy head to reward staying at the goal, but it still keeps SRL’s dwell requirement of k consecutive steps. The paper does not explain why both mechanisms are needed. The ablation (k = 1, with the occupancy head) is missing, so it is unclear whether the occupancy head can replace the dwell objective or only complements it. And the occupancy head matters only on Humanoid. It seems that the occupancy value contributes to the policy objective essentially only when the agent falls.
  3. Several configuration changes are not ablated and may account for a large part of the gain. Besides geometric losses and occupancy head, Table 1 lists more differences from SRL. None of these is ablated individually. For example, for Ant Hardest Maze task, the results are: SRL (143), ChronoSRL with one-step actions (486), full ChronoSRL(682), w/o geometric losses (647), w/o occupancy head (681). It seems that improvement comes from these unablated settings.
Questions for authors
  1. Could the author clearly explain the role of the geometric losses according to the argument of Weakness 1?
  2. Regarding the occupancy head, could the author provide an ablation study (k = 1, with the occupancy head)? Could the author explain for different scenarios (eg., the agent falling, drifting away or overshooting with too much momentum), how or whether the occupancy head helps?
  3. Could the author provide more ablation studies about the modified parts compared to SRL? Especially, I am interested in these two settings: (1). ChronoSRL without geometric losses, without occupancy head; (2). SRL only with geometric losses and occupancy head.
2 comments

Thanks a lot for the careful review!

  1. You asked us to clearly explain the role of the geometric losses. You are right that they act as an auxiliary loss with an interpretable target, since they shape the embeddings from which both heads predict the value. Their effect on control is faster learning. They speed up early learning in nearly every ant-maze setting (new Table 5), and in Ant U4-Maze, where the fastest route is not the straight line, the variants whose distance follows the time learn fastest (new Table 8). The permuted variant is one of these fast variants, because it only shuffles the targets of the time loss and its distance still follows the time. The losses also do not only help a weak critic, as they still raise the AUC of SRL with 16 layers, where SRL is much stronger on its own (new Table 6). The Humanoid failure of SRL with our losses comes from SRL's layer-normalized embeddings, and with ChronoSRL's embeddings, it learns with all four seeds (Appendix F.1).

  2. We ran the k = 1 ablation you asked for (Appendix F.2). On Humanoid, the occupancy head alone and the dwell sequences alone each keep the humanoid at the goal for about 50 to 60% of the episode, and both together for 83%. With neither, the humanoid falls right after reaching the goal, so the two mechanisms complement each other. We also rolled out the variant without the head for your scenarios. It never overshoots and almost never drifts away, but after standing at the goal for about 400 steps, it falls in 75% of the episodes in which it reaches the goal, against 23% for ChronoSRL. This also corrects a sentence of our submission, which said that the humanoid overshoots. You are right that the occupancy head matters most when the agent can fall, and this is by design. The quadruped can fall as well, and without the head, its final time at the goal drops from 893 to 55 for velocity tracking (new column in Table 4).

  3. We ran both settings you were interested in (new rows in Table 3) and also added all other settings to SRL one group at a time (new Table 7). In the ant mazes, these settings bring most of the gain over SRL, and with action chunks alone, the survival critic of SRL already learns faster than AC-CRL, which also uses chunks. On Humanoid, the settings help little, and the occupancy head and the geometric losses do most of the work. Adding both components to SRL in its own configuration raises its final time at the goal on Humanoid from 49% to 79% of the episode.

We hope that this answers your questions, and we would be very happy if you would consider raising your score. Thanks again for the detailed suggestions, which really improved the paper!

Thanks for the rebuttal comments and additional experiments. My questions have been answered:

  1. The geometric losses work as an extra (auxiliary) loss. They help the agent learn faster early in training (Table 5), but they do not change the final performance much. I think this is a fair design choice and a real contribution. The authors also give a convincing reason why the geometric losses fail on Humanoid.
  2. The authors explain how the occupancy head and the k-step dwell rule work together. From the study of how the humanoid fails and the ablations in Table 3, I think the occupancy head mainly helps by keeping the robot from falling.

I thank the authors for their clear answers and honest experiments, and I will raise my soundness score. However, these answers also make the contribution of the paper smaller. As I said above, the title, abstract, and Figure 1 present the geometric losses as the main idea, but they mainly act as an extra loss that speeds up early learning (about 12% higher early AUC, as a median). They have little effect on final performance. The occupancy head clearly works, but mainly in tasks where the agent can fall. In the ant mazes, most of the gain over SRL comes from action chunks and other setting changes. I suggest that the authors change the abstract and introduction so that their claims match these results. Also, the two parts are mostly independent of each other. So the paper reads more like two separate, small improvements to SRL than one connected paper built around temporal geometry.

To be honest, I find it hard to decide on my score. I see this as a borderline paper, and I will keep my score at 2, although I do not think my exact score matters much. I no longer have concerns about the technical soundness, and the experiments are careful and honest. But I think the method’s contribution is either just below the bar for ICLR or only barely above it. If the other reviewers and the AC think the contribution is enough, I am fine with that and will not argue against accepting the paper.

Reviewer_jTmeOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ChronoSRL, which is a self-supervised goal-conditioned RL method that gives the critic’s representation an explicit temporal geometry, so that the distance between state–action and goal embeddings reflects how long the agent takes to reach the goal. Together with four kinds of losses in total, ChronoSRL learns faster and reaches higher time-at-goal than other baselines on navigation and locomotion tasks. The temporal geometry is learned in maze environment, and it is able to extend to realistic robot learning.

Critical strengths

The experimental evaluation is comprehensive and multi-faceted. The paper moves from an overall comparison against strong baselines, to component ablations, to several behavioral analyses of the learned geometry and goal-reaching behavior, and finally to realistic quadruped locomotion tasks, which together give a clear picture of which components contribute and how.

Critical weaknesses

The results seem suggest that most of the performance gain comes from the occupancy head rather than from the temporal geometry that the paper is built around. Section 4.3 shows that the geometry is learned well, but Table 3 shows its contribution to performance is marginal, so the claimed contribution is not consistent with the evidence. As a result, the paper reads as a technically strong engineering effort whose central scientific claim is not supported by its own ablations.

Questions for authors

None

1 comment

Thanks a lot for the review, and for appreciating how comprehensive our evaluation is!

Your main concern was that most of the gain seems to come from the occupancy head rather than from the temporal geometry. We want to highlight that the two components do different jobs, and the revised paper now states the role of each part in Section 4.2 and the Discussion. The occupancy head keeps the agent at the goal, so it matters most where the agent can fall, like the humanoid and the quadruped. The geometric losses make learning faster. When we add them to SRL, they raise its AUC in every setting we tried (new Table 6), and on Ant Hardest Maze with 8 layers they even more than double it. Inside ChronoSRL, they speed up early learning in nearly every ant-maze setting (new Table 5). This speedup is linked to the temporal geometry, as in Ant U4-Maze the variants whose distance follows the time are the ones that learn fastest (new Table 8).

Overall, ChronoSRL learns faster than the strongest of SRL, CRL and AC-CRL in 34 of the 35 settings where any method learns (new Appendix J).

We hope that this makes the contribution of the temporal geometry clearer, and we would be very happy if you would consider raising your score. Thank you again for the review!

Reviewer_cqQzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ChronoSRL, which adds temporal-distance supervision to a survival-based goal-conditioned critic. Shared embeddings predict both goal-arrival times and future goal occupancy, and these predictions guide an action-chunked policy. Experiments show improved learning efficiency on seven locomotion and navigation benchmarks, with additional evaluations on simulated quadruped tasks using shaping costs.

Critical strengths
  1. The empirical improvements are substantial across tasks and network depths. ChronoSRL often matches much deeper baselines with fewer environment steps, making the gains practically meaningful.

  2. Predicting goal occupancy addresses an important limitation of arrival-only objectives: reaching a goal does not guarantee staying there. The humanoid ablation provides clear evidence that this component prevents falls after arrival and improves sustained goal achievement.

Critical weaknesses
  1. The contribution of accurate temporal supervision is less convincing than the overall performance gains. In Table 3, removing both geometric losses preserves most of the improvement, while shuffling the time targets gives almost unchanged AUC on Ant Hardest Maze (674 versus 682) and Ant U4-Maze (800 versus 801). These results leave it unclear how much benefit comes specifically from learning temporal geometry, beyond the other changes to SRL.

  2. The separation objective weakens the interpretation of distance as a measure of time. Cross-batch goals are pushed at least one observation horizon away even when they may be quickly reachable. Appendix G.2 also reports several-fold overestimation of remaining time. The evidence therefore supports temporal ordering more clearly than an accurately calibrated temporal distance, which limits a central claim of the paper.

Questions for authors
  1. How much do the geometric losses contribute to the quadruped results when all other components are fixed? Does using correctly paired time targets outperform shuffled targets there?

  2. How should the claimed temporal interpretation be understood given the overestimation in Appendix G.2? Would restricting separation to genuinely censored pairs retain the learning benefits while improving calibration?

1 comment

Thanks a lot for the thoughtful review, and for pointing out that the gains are practically meaningful and that the humanoid ablation shows how the occupancy head prevents falls after arrival!

  1. You were concerned that the permuted times reach almost the same AUC as ChronoSRL, so it is unclear how much the temporal geometry contributes. We want to highlight that permuting the times does not remove the temporal geometry. It only shuffles the targets of the time loss, while the survival likelihood and the separation loss still use the measured times. So the distance of this variant still follows the time as closely as that of ChronoSRL, and it also learns about as fast. To test the role of the geometry directly, we compared the geometry and the learning speed of every variant of our losses (new Table 8). In Ant U4-Maze, where the fastest route is not the straight line, the variants whose distance follows the time are the ones that learn fastest. Across the ant mazes, the losses also speed up early learning in nearly every setting (new Table 5).

  2. You are right that the separation loss stretches the scale of the distance. We ran the variant you suggested, with separation only on the censored pairs (Appendix G.2). It learns as fast as ChronoSRL on Ant Hardest Maze and Ant U4-Maze and somewhat slower on Humanoid, but the overestimate stays, so the cross-batch pairs do not cause it. Without any separation loss, the overestimate roughly halves, but in Ant U4-Maze the distance then follows the straight line instead of the time. So the separation loss is what keeps the distance ordered by time, at the cost of a stretched scale, and the revised paper now states this clearly. What the policy acts on is much closer to the real times, though. It never reads the distance directly, and the hazard value it uses is only 11 to 16% longer than the waiting time the agent actually experiences (Appendix G.4).

  3. For Go2, we trained all three tasks with 16 layers without the geometric losses and with permuted times (new Appendix H.4). Neither variant changes the final performance significantly, so on the quadruped the losses are not needed for the final results, while they speed up learning on the benchmarks. What Go2 does need is the occupancy head. Without it, the robot falls early, and its final time at the goal drops from 893 to 55 for velocity tracking (new column in Table 4).

We hope that this clears up the role of the temporal supervision, and we would be very happy if you would consider raising your score. Thanks again for the helpful suggestions, especially the experiment with the censored pairs!

Reviewer_HTGAOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a goal-conditioned RL approach, which builds on prior work that learns representations capturing temporal distance and uses those for control. The authors use concepts from survival RL to account not only for the ability to reach the goal or minimize heading time but also to stay at the goal once you get there. The authors claim that this combined objective, along with implementation decisions like action chunking, substantially can improve performance on standard locomotion RL benchmarks (JaxGCRL).

Critical strengths
  • Strong performance on JaxGCRL, outperforming prior approaches like CRL.
  • Expounds on the subtle distinction between "reaching" vs. "staying" at the goal in GCRL, i.e., hitting times vs. discounted rewards as well as temporal distance vs. spatial distance.
Critical weaknesses
  • Given the temporal distance framing, it would be valuable to compare against prior temporal-distance-focused methods, in particular quasimetric methods that explicitly capture the asymmetric inductive biases of these distances [1, 2].
  • Missing evaluation and discussion of temporal distance learning methods; see questions. Why do we want to fit a symmetric Euclidean distance here?
  • No formal significance testing is reported.
  • Despite being motivated by a learning-environment-based temporally accurate distance, they need to use epsilon balls to compute goal reaching. This then introduces a dependence on a ground-truth Euclidean or similar hard-coded metric from the environment. This is fine for locomotion but becomes a limitation with a larger manipulation environment.

I would like to raise my score if these concerns can be fully addressed.

References

[1] Liu, B. et al., 2023. "Metric Residual Network for Sample Efficient Goal-Conditioned Reinforcement Learning." AAAI

[2] Wang, T. et al., 2023. "Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning." ICML

Questions for authors
  • Could you briefly describe the justification for learning a policy specifically to stay at the goal instead of just reaching it? In theory, we can always find optimal stationary policies in an MDP. In other words, the policy doesn't need to change its behavior to stay at the goal once it reaches it, in theory. If this were the case, then the only difference between an optimal goal-reaching policy and a goal-staying-and-reaching policy would be the action taken at the single state, which is the goal. This could be easily learned or hard-coded with some separate policy. Why do we need to overhaul the entire objective to handle this goal-staying problem? If we only focus on goal reaching/hitting time, like in CMD [1], we don't need to worry about that other term at all.
  • What bias is introduced by using Equation 2 to regress onto a probability? Is it an appropriate Bregman divergence with the logits used? How would it compare to BCE or LINEX losses [2] in terms of grad stability?
  • Other related work? TRA [3], Ye et al. [4], DWSL [5]. Please discuss and compare with this work.
  • Evaluations are heavily skewed towards locomotion/navigation tasks. These are the tasks where the task topology is locally Euclidean, and thus you expect distance-learning-based methods (in particular ones that don't capture asymmetry) to do fine. The paper would benefit from environments with more stochasticity, high-dimensional interactions, and irreversibility. There are pushing experiments, but these also are fairly low-dimensional in structure, since they often can reduce to learning Euclidean distances for end-effector control.
  • Minor: code and implementation details would be helpful.
  • Could we improve the "staying at the goal" problem if we included extra state information in the goal? In your example of the humanoid, it seems that including velocity information would work.
  • Clarify what you mean by shaping when you learn the distances. How is this different from value learning?

References

[1] Myers, V. et al., 2024. "Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making." ICML

[2] Parsian, A. and Kirmani, S., 2002. "Estimation Under LINEX Loss Function." Handbook of Applied Econometrics and Statistical Inference

[3] Myers, V. et al., 2025. "Temporal Representation Alignment: Successor Features Enable Emergent Compositionality in Robot Instruction Following." NeurIPS

[4] Ye, J. et al., 2021. "Auxiliary Tasks Speed Up Learning Point Goal Navigation." CoRL

[5] Hejna, J. et al., 2023. "Distance Weighted Supervised Learning for Offline Interaction Data." ICML

1 comment

Thanks a lot for the detailed review and for all the pointers to related work! You wrote that you would raise your score if your concerns are addressed, so we tried to address every one of them with new experiments or changes in the paper.

  1. You asked us to compare against quasimetric methods and why we fit a symmetric distance. Our distance is actually not symmetric, since the state-action encoder and the goal encoder are two separate networks. Along trajectories, the distance is larger against the direction of motion in most pairs (new Appendix G.3). We also replaced our distance with the MRN quasimetric of Liu et al. It does not help our critic, and when we use it in CRL and AC-CRL, which is essentially the critic of CMD, it makes them worse (new rows in Table 3). QRL needs its own objective and a learned dynamics model, so we were not able to port it faithfully within the discussion period.

  2. For significance, we added tests for all comparisons (new Appendix J). ChronoSRL is better than the strongest baseline in 34 of the 35 settings where any method learns, significantly in 33 of them, and it is never significantly worse. Even after correcting for multiple comparisons, 31 of them stay significant.

  3. You are right that ChronoSRL, like SRL, relies on a region around each goal. Besides telling when a goal is reached, we also use it for the occupancy head and the cross-batch goals (Appendix B), and we now name this as a limitation in Section 5. The distance itself, however, follows the time the agent needs and not the Euclidean distance (Section 4.3).

  4. On staying versus reaching, we agree that an optimal stationary policy exists. But a hitting-time objective stops at the first arrival, so it never trains what happens afterwards, and for a humanoid or a quadruped, staying means balancing at every step and not just one action at the goal. We tested this directly with single goals and without the occupancy head. The humanoid then reaches the goal in almost every episode but falls right after, so it spends less than 1% of the episode at the goal, against 83% for ChronoSRL (Appendix F.2).

  5. Equation 2 does not regress onto a probability. It regresses the distance onto the measured time (scaled by κ\kappa) with a Huber loss, so it targets about the expected time and its gradient stays bounded. A BCE loss on e−de^{-d} would instead converge to −log⁡E[γτ]-\log E[\gamma^\tau], which underestimates the expected time by Jensen's inequality, and its gradient grows without bound as dd goes to zero. A LINEX loss with unit scale has the same target as this BCE loss, while Bregman divergences on the times, such as the squared loss, target the expected time just like ours. We now state this in Section 3.2.

  6. Thanks for the pointers to TRA, Ye et al. and DWSL, we added all three to the related work.

  7. On the tasks, the humanoid and the quadruped can fall, which ends the episode for good, so both are irreversible. Go2 also trains with randomized physics and observation noise (Appendix H). We agree that manipulation is still an open problem for survival critics (Appendix I).

  8. The anonymized code and the policies are available at https://github.com/chronosrl/chronosrl and https://chronosrl.github.io.

  9. Adding velocity to the goal would unfortunately not be enough. Without the occupancy head, the humanoid stands at the goal for hundreds of steps before it falls, so a goal with zero velocity would already be met long before the fall. The occupancy head instead values every step after the arrival (Appendix F.2).

  10. With shaping, we mean cost terms such as energy or smoothness that we add to the measured times, so a costly path counts as a longer one (Section 3.4). This is close to Monte Carlo value learning, since the critic is fit directly to these measured times, so no shaped reward is ever bootstrapped.

We hope that we could address all your concerns, and we would be very happy if you would consider raising your score. Thank you again for the helpful review!

Comments on the paper

1 comment

Thanks to all reviewers for the reviews! You all found the empirical gains strong, and the main question was what each part of ChronoSRL contributes. We were able to ran almost all experiments you asked for and all new / updated text in the PDF is blue.

Here are the highlights of the most important points. More specific answers to each review are in the rebuttals for each reviewer.

  1. One big concern were the impact of our geometric losses. We want to highlight that the losses make learning faster. They speed up early learning in nearly every ant-maze setting (new Table 5). When we add them to SRL, they raise its AUC in every setting we tried (new Table 6). This speedup comes directly from the temporal geometry. In Ant U4-Maze, where the fastest route is not the straight line, learning is fastest (through our losses) when the distance follows the time (new Table 8).

  2. Similarly, the reviewers were asking about the necessity of the occupancy head. We want to highlight how the occupancy head keeps the agent at the goal. Without it, the humanoid often falls after reaching the goal (Appendix F). Go2 even falls early, and its final time at the goal drops from 893 to 55 (new column in Table 4).

Overall, ChronoSRL learns faster than the strongest of SRL, CRL and AC-CRL in 34 of the 35 settings and it is never worse based on new significance tests in Appendix J. We also uploaded the anonymized code at https://github.com/chronosrl/chronosrl, and our policies can be tested in the browser at https://chronosrl.github.io.

We hope that we could show that our method is not only significantly better in terms of end performance, learning speed and empirical sample efficiency but also that all the components that we add are important for that and can also be added to the baseline algorithms, like SRL, to improve their performance as well. While our method might not be the most fancy theoretically, we hope that the mentioned points highlight why we think that ChronoSRL and also the new robotics environments we added, are a great addition to the field.

Thank you again for reviewing and improving our paper with your suggestions!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.