acceptodds
Withdrawn from ICLR 2027

Auxiliary Midpoint Learning for Goal-Reaching Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We propose *Auxiliary Midpoint Learning* (AML), a technique for quasi-metric learning in goal-reaching reinforcement learning (GRRL). In addition to standard temporal-difference (TD) learning of the policy and quasi-metric networks, AML jointly trains a network that predicts midpoints with respect to the learned quasi-metric and uses these predictions to improve the global consistency of the quasi-metric network. This enables more effective learning on goal-reaching tasks with long horizons. Experiments show that AML consistently improves over the quasi-metric TD baseline and achieves the highest success rate among the compared methods on several benchmark environments.

Primary area:
reinforcement learning
Keywords:
Goal-Reaching Reinforcement Learning, Quasi-metric Learning, Midpoint Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_G1ixOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes auxiliary midpoint learning (AML) for long-horizon goal-reaching RL, in which the goal is to learn a policy that can reach any given goal state from a starting state. The main idea is learning to predict an intermediate state / midpoint between the current state and the goal, which can help with improving the learned quasi-metric value function. Unlike standard TD learning which updates the value function using neighbouring states, AML uses the predicted midpoints to propagate information over longer distances. The paper shows that across six different environments it consistently improves or matches a TD baseline. The improvements are more noticable in the larger maze environment.

Critical strengths
  1. The paper identifies a limitation of TD learning for long-horizon tasks, and the main idea is well motivated.

  2. AML improves over the quasi-metric TD baseline across the evaluated environments, especially on Ant Big Maze (RS). This shows that midpoint learning could be useful for tasks that might have a larger state space, or a long-horizon reasoning about the state space.

  3. The proposed approach adds an auxiliary midpoint learning objective which seems easy to add to the general goal-reaching RL framework, and seems to be applicable to any other similar methods.

Critical weaknesses
  1. The proposed idea has limited novelty in comparison to prior work, especially Kasaura et al. The main contribution of the paper seems to be adapting the approach of that paper (enforcing global consistency of a quasi-metric) to goal-reaching RL. While this is potentially valuable, I think the paper could benefit from a discussion of the technical challenges specific to the RL setting and which aspects of the proposed algorithm are necessary to address them.

  2. There are a few important ablations missing from the paper. Specifically, 1) if we replace the midpoints by simpler states (e.g. random states along sampled trajectories), how do the results change? 2) Is the discriminator necessary? 3) How sensitive are the results to the weight of the additional objective?

  3. The main motivation of the paper seems to be that TD learning can underestimate distances between states far from each other, and that midpoint learning can especially be useful for those settings. However, the experiments mainly focus on / evaluate policy success rather than the accuracy of the learned value function. I think this work could benefit from additional evals. For example, in a simple maze setup where the shortest distances can be computed, you could measure how similar the learned value function is to the ground-truth distance.

  4. The paper is difficult to read, and lacks visualizations that can build intuition, e.g. visualizations of predicted midpoints.

Questions for authors
  1. Why is minimizing the max of the two distances preferable to minimizing their difference or directly encouraging the two distances to be equal?

  2. Have you tried any ablations suggested in weakness #2? e.g. replacing midpoints with simpler states, such as states sampled from previous trajectories?

  3. What challenges do you expect when applying AML to more complex environments?

Reviewer_N2nHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Auxiliary Midpoint Learning (AML) for goal-reaching RL with quasi-metric value functions. Next to standard TD learning, a network predicts the midpoint between a state and a goal, and the value function is trained to be consistent with it, i.e. the distance to the goal should equal the sum of the distances via the midpoint. The motivation is a theorem adapted from Kasaura (2025) stating that midpoint consistency plus local accuracy implies global accuracy. Experiments on six JaxGCRL tasks show improvements over the authors' TD baseline, in particular in a long-horizon maze.

Critical strengths
  • Simple plug-in mechanism against the underestimation of long-range distances in quasi-metric value functions, which could be added to any quasi-metric GCRL method.
  • Clear gain over the TD baseline in the long-horizon Ant Big Maze (RS) (Fig. 3: about 0.9 vs. 0.6 success).
Critical weaknesses
  • Closest prior work and baselines are missing, so novelty is overstated.
    • RIS (Chane-Sane et al., 2021) already learns a subgoal network for the point halfway between ss and gg with max⁡(V(s,sg),V(sg,g))\max(V(s,s_g),V(s_g,g)) on replay states, in online GCRL. The only difference: RIS uses it for the policy, AML for VV.
    • Transitive RL (Park et al., 2026) uses the triangle inequality as value backup, with the same O(log⁡T)O(\log T) motivation. It shows that a learned subgoal generator (COE, Piekos et al., 2023), very similar to AML's midpoint network, overestimates, while in-trajectory subgoals work. TRL is offline, but in-trajectory subgoals can be taken from a replay buffer as well. So what does the learned midpoint network add?
    • DC-MCTS (Parascandolo et al., 2020) and GCP (Pertsch et al., 2020) also predict midpoints recursively.
    • None of these is cited. RIS and an in-trajectory-subgoal variant should be baselines. The QRL re-implementation (near zero in 5/6 tasks, L451-453) is not informative.
  • The theory does not apply, and the experiments do not test the mechanism.
    • Theorem 1 requires a midpoint for every pair, a uniformly continuous mm and locally accurate VV. The paper itself says that mm can be discontinuous (L474-475) and that VV is not meant to be accurate (App. B). So the theorem is motivation only, and "theoretically, this approach also prevents the underestimation" (L50-51) overclaims.
    • Since VV is a quasi-metric, V(s,m)+V(m,g)≥V(s,g)V(s,m)+V(m,g)\ge V(s,g) for any mm. The midpoint term can only increase VV. If mm is off the shortest path (or with stochastic dynamics), this causes overestimation, which is exactly the failure mode reported for learned subgoal generators in TRL. This is not analysed.
    • The claims "improves global consistency" and "fewer iterations than TD" are not measured: no comparison of VV to ground-truth maze distances, no visualisation of midpoints, no ablations (discriminator, cmc_m, log loss, max⁡\max vs. sum-of-squares).
    • 5 seeds in four of six tasks, only AML/Baseline were tuned (App. E), and AML is clearly better than SAC+HER/CRL in only 2/6 tasks.
Questions for authors
  1. How does AML differ from RIS beyond using the subgoals for VV instead of the policy? Can you add RIS as a baseline, and an ablation where the learned midpoint is replaced by an in-trajectory state between ss and gg from the replay buffer (as in TRL)?
  2. Is there direct evidence for the mechanism, e.g. learned V(s,g)V(s,g) vs. ground-truth shortest-path distance in Ant Big Maze, and predicted midpoints decoded to (x,y)(x,y)?
  3. Do you observe overestimation (e.g. VV hitting Vmax⁡V_{\max})? What happens without the discriminator?
  4. Is the advantage robust when the other methods are tuned as well? Please report final success with confidence intervals and more seeds.
  5. What is c(a)c(a) in the experiments?

Suggestions:

  • Evaluate on OGBench giant mazes, where long-horizon value accuracy matters most.

Minor points / Details

  • L36-37: "QRL is expected to converge faster": please give evidence.
  • L114-119: the argument about CRL's limited representation is hand-wavy.
  • L183-184: in stochastic MDPs the midpoint equality (Eq. 3) generally does not hold; please state the deterministic assumption.
  • Eq. 4 takes the argmin over the latent space, Theorem 1 over XX; in practice this is only approximated through DD.
  • Eq. 5: VmV_m uses the online and not the target networks. Intended?
  • Sec. 4.2: HER uses only the final state for relabelling instead of future-state sampling. Was SAC+HER run with its default?
  • App. A: ε<1/3\varepsilon<1/3 should be stated in the theorem. The result is essentially Proposition 2 of Kasaura (2025).
  • App. C: the argument for max⁡\max over sum-of-squares is only qualitative.
  • App. E: a replay buffer of 10410^4 is very small for 10810^8 steps with 512 environments. Please confirm.
  • L384: "IQE-maxmean has stronger theoretical guarantees": please cite.
  • Missing related work, most relevant:
    • E. Chane-Sane, C. Schmid, I. Laptev. Goal-Conditioned Reinforcement Learning with Imagined Subgoals. ICML 2021. (Closest prior work.)
    • S. Park, A. Oberai, P. Atreya, S. Levine. Transitive RL: Value Learning via Divide and Conquer. ICLR 2026. (Triangle-inequality value backup with in-trajectory subgoals.)
    • P. Piekos, A. Ramesh, F. Faccio, J. Schmidhuber. Efficient Value Propagation with the Compositional Optimality Equation. NeurIPS Workshop on Goal-Conditioned RL 2023. (Learned subgoal generator for triangle-inequality updates.)
    • G. Parascandolo, L. Buesing, J. Merel, L. Hasenclever, J. Aslanides, J. B. Hamrick, N. Heess, A. Neitz, T. Weber. Divide-and-Conquer Monte Carlo Tree Search for Goal-Directed Planning. arXiv:2004.11410, 2020. (Recursive midpoint proposals.)
    • K. Pertsch, O. Rybkin, F. Ebert, C. Finn, D. Jayaraman, S. Levine. Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors. NeurIPS 2020. (Recursive midpoint prediction.)
    • C. Pinneri, G. Martius, A. Krause. Neural All-Pairs Shortest Path for Reinforcement Learning. NeurIPS Deep RL Workshop 2022. (Triangle-based correction of underestimated distances with intermediate replay states.)
Reviewer_tpZ7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Auxiliary Midpoint Learning (AML) for online goal-reaching reinforcement learning. Alongside TD learning of a quasimetric value function and policy, it trains a network to predict a midpoint between the starting state and the goal. The sum of the predicted costs of the two segments through this midpoint serves as an auxiliary target for the overall value, with the aim of improving long-horizon value learning. The midpoint network minimizes the larger of the two segment costs, while a discriminator encourages generated latents to resemble representations of actual states. The paper evaluates AML in six online continuous-control environments and reports improvements over a quasimetric TD baseline without midpoint auxiliary learning.

Critical strengths
  1. A clear motivation for improving long-horizon value learning. Augmenting one-step TD learning with cost-composition signals through midpoints is an intuitive idea. The paper presents a concrete approach to incorporating existing midpoint-learning principles into online goal reaching. The conditional theorem in Appendix A provides mathematical motivation for this design.

  2. A direct baseline for evaluating the auxiliary module. The paper includes a (c_m=0) baseline that removes midpoint auxiliary learning from the same implementation. This is a useful controlled comparison for assessing the practical benefit of the additional module.

Critical weaknesses
  1. The validity of the learned midpoints and their role in policy improvement are insufficiently validated.

The method trains midpoints using the current value function and then uses values evaluated through those midpoints to supervise the value function itself. It is therefore important to establish that the generated midpoints correspond to valid environment states and appropriate intermediates between the specified start and goal. However, the manuscript provides no visualizations or quantitative evaluations demonstrating this correspondence, the reachability of the two segments, or the balance of their actual costs. The discriminator distinguishes generated latents from representations of observed states. It does not assess midpoint suitability conditioned on the start and goal, so similarity to the state distribution does not establish that a generated latent is an appropriate intermediate. The theorem in Appendix A also assumes local distance accuracy, exact midpoint optimization, and consistency conditions. Whether these assumptions hold during the actual joint learning process remains unverified.

Furthermore, distance underestimation or additive inconsistency does not necessarily imply incorrect action selection. For example, uniform positive rescaling can preserve action rankings. Improving distance consistency should therefore be distinguished from improving the action rankings or update directions relevant to policy learning. Appendix B similarly acknowledges that the practical role of the learned value function may be to support policy learning rather than accurately reconstruct true costs.

The existing ablation shows that adding midpoint auxiliary learning can improve policy success. However, it does not sufficiently establish whether learning valid midpoints and correcting long-range value errors are the mechanisms responsible for that improvement.

  1. The incremental contribution over existing midpoint methods and the advantages over closely related alternatives are not sufficiently isolated.

Quasimetric learning for online goal reaching already appears in QRL, and jointly learning midpoints and distance functions was proposed in Kasaura’s midpoint-learning work. The authors cite these studies and explain the differences. Nevertheless, the current comparisons make it difficult to assess the additional benefits of the proposed midpoint-generation procedure and objective.

The (c_m=0) baseline demonstrates the utility of the auxiliary module as a whole. It does not establish whether predicted midpoints are more useful than intermediate states drawn from observed trajectories, or whether the proposed minimax objective offers an advantage over the existing squared-sum objective.

Questions for authors
  1. Can you establish that the predicted midpoints are valid intermediate states? In representative environments, please explain the correspondence between midpoint latents and actual states, and evaluate whether the midpoint is reachable from the start and whether the goal is reachable from the midpoint. Evidence of balanced segment costs that is independent of the current value function or discriminator would be particularly helpful.

  2. Is there direct evidence that midpoint learning improves values in ways that benefit policy decisions? Could you compare long-range value errors and action-ranking accuracy against actual rollout outcomes, with and without midpoint auxiliary learning? This would help distinguish numerical distance consistency from improved decision-making and clarify the source of the reported success-rate gains.

  3. What advantages remain over closely related midpoint or intermediate-state alternatives? On a small representative subset of environments, could you hold the architecture and training budget fixed while replacing predicted midpoints with intermediate states from replay trajectories, or using the existing squared-sum midpoint objective? These controls would help distinguish the benefit of additional auxiliary supervision from the benefit of the proposed midpoint design itself.

Reviewer_C3wuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Goal-conditioned reinforcement learning (GCRL) learns value functions to reach specific goals that correspond to hitting times or temporal distances. While classical methods for GCRL either view it as a multitask RL problem with hindsight relabeling or try and exploit the inductive biases of distances through causal metric architectural choices, this paper integrates these two approaches. AML augments the quasi-metric architecture with explicit waypoint consistency. Compelling results are presented on some locomotion environments, including Ant Maze.

Critical strengths

Strong online performance with a method based on quasi-metric architectures.

The extra waypoint objectives appear to reduce the instability of restricting to a quasi-metric architecture.

Critical weaknesses

Although technically the state spaces are all high-dimensional, the actual task-level geometry is usually locally low-dimensional: either navigation or essentially moving an end effector in low dimensions with or without one other object. Testing higher-dimensional and/or visual environments would clearly show if the method effectively scales.

Missing related work: Transitive RL [1], TMD [2], Horizon Generalization [3], and RIS [4]. These works address temporal-distance learning, waypoint-based value learning, or imagined subgoals, and should be contrasted with AML's midpoint and TD objectives.

Questions for authors
  • Could you compare against related methods based on quasi-metric representations and values, and particularly those that are used online? Comparisons against methods like MRN [5] and CMD [6] would be helpful, along with a methodological comparison to offline TMD [2]. It would also be helpful to contrast with Transitive RL's waypoint-based value updates [1], Horizon Generalization's planning-invariance analysis [3], and RIS's learned imagined subgoals [4].

  • Comment on the ability of your approach to handle stochasticity in the environment. Classically, this is an issue when you view GCRL as a shortest-distance problem, as methods like QRL [8] do. This works in a deterministic environment where optimal goal reaching is equivalent to graph shortest path, but fails in stochastic environments where you typically need to do a log-sum-exp over a family of paths. In stochastic environments, you need an unbiased learning procedure to handle the stochastic dynamics, and distinguish dynamics noise from action noise, independently of whether the transition model is stochastic or not. Not accounting for this means that methods like QRL will be too optimistic due to maximization bias over actions. This will occur when the dynamics are stochastic or even if there's uncertainty about the dynamics, resulting in epistemic stochasticity, such as in visual environments, which are always inherently a bit noisily observed. It makes methods like QRL struggle there. Approaches like TMD [2] do do this but at the cost of requiring TD-style updates along with the quasi-metric parameterization, while approaches like MRN [5], CMD [6], and MQE [7] don't require this but unfortunately sacrifice the ability to learn optimal policies off-policy.

Goal-Reaching Reinforcement Learning (GRRL) (Wang et al., 2023), also simply called Goal-Conditioned Reinforcement Learning (GCRL) in some works [10]

Minor: could you explain why the term GRRL is used and where that acronym originated? GCRL appears to be the more standard term; earlier work also uses goal-conditioned value functions [9]. I also hear "goal reaching" just as two words used often, but "GRRL" seems less common. I'm curious if you're aware of some literature where that acronym is more standard.

References

[1] Seohong Park, Aditya Oberai, Pranav Atreya, and Sergey Levine. Transitive RL: Value Learning via Divide and Conquer. ICLR, 2026. https://arxiv.org/abs/2510.22512

[2] Vivek Myers, Bill Chunyuan Zheng, Benjamin Eysenbach, and Sergey Levine. Offline Goal-conditioned Reinforcement Learning with Quasimetric Representations. NeurIPS, 2025. https://arxiv.org/abs/2509.20478

[3] Vivek Myers, Catherine Ji, and Benjamin Eysenbach. Horizon Generalization in Reinforcement Learning. ICLR, 2025. https://arxiv.org/abs/2501.02709

[4] Elliot Chane-Sane, Cordelia Schmid, and Ivan Laptev. Goal-Conditioned Reinforcement Learning with Imagined Subgoals. ICML, 2021. https://proceedings.mlr.press/v139/chane-sane21a.html

[5] Bo Liu, Yihao Feng, Qiang Liu, and Peter Stone. Metric Residual Network for Sample Efficient Goal-Conditioned Reinforcement Learning. AAAI, 2023. https://arxiv.org/abs/2208.08133

[6] Vivek Myers, Chongyi Zheng, Anca Dragan, Sergey Levine, and Benjamin Eysenbach. Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making. ICML, 2024. https://arxiv.org/abs/2406.17098

[7] Bill Chunyuan Zheng, Vivek Myers, Benjamin Eysenbach, and Sergey Levine. Multistep Quasimetric Learning for Scalable Goal-conditioned Reinforcement Learning. arXiv:2511.07730, 2025. https://arxiv.org/abs/2511.07730

[8] Tongzhou Wang, Antonio Torralba, Phillip Isola, and Amy Zhang. Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning. ICML, 2023. https://proceedings.mlr.press/v202/wang23al.html

[9] Vikas Dhiman, Shurjo Banerjee, Jeffrey M. Siskind, and Jason J. Corso. Floyd-Warshall Reinforcement Learning: Learning from Past Experiences to Reach New Goals. arXiv:1809.09318, 2018 (revised 2019). https://arxiv.org/abs/1809.09318

[10] Michał Bortkiewicz, Władysław Pałucki, Vivek Myers, Tadeusz Dziarmaga, Tomasz Arczewski, Łukasz Kuciński, and Benjamin Eysenbach. Accelerating Goal-Conditioned Reinforcement Learning Algorithms and Research. ICLR, 2025. https://arxiv.org/abs/2408.11052

Comments on the paper

1 comment
AuthorsOpenReview ↗

Update and Withdrawal

Thank all reviewers to your thoughtful and informative reviews!

To keep record it, we update our paper as we mentioned as our comment. We expanded related work sections to include missing prior research. We also added four Appendix:

  • Appendix G: Visualization of examples of learned V-values and predicted midpoints. This suggests that our method can predict effective intermediate points and they appear to be contributing to the value function as we intended.
  • Appendix H: Visulization of predicted midpoints in our method and ablation studies: a method which minimizes the sum of squares and a method without discriminator. This suggests that our change of objective function and the loss term from discriminator appear to be work as we intended.
  • Appendix J: The additional experiments for Online TRL and RIS in Ant U-Maze (RS) and Ant Big Maze (RS) environments.
  • Appendix K: The statistical test for the results in main text.

However, we decided to withdraw our paper according to the results of additional experiment, shown in Appendix J. While our method outperformed RIS in Ant Big Maze (RS) environment, Online TRL, which samples intermediate points from past trajectories and uses them for update of V values instead of predicted midpoints, showed comparable performance to our method. These results raise questions about whether the effectiveness of the proposed method stems from the prediction of the midpoint. While these results themselves may be interesting and it remains possible that our method’s superiority will be demonstrated in a larger environment, such as OGBench giant maze suggested by Reviewer N2nH, we withdraw this paper because the paper's argument cannot be upheld as is. We aim to write a better paper based on further research.

We have learned a great many things about this field through reviews. We would like to express our gratitude to reviews once again.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.