- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes auxiliary midpoint learning (AML) for long-horizon goal-reaching RL, in which the goal is to learn a policy that can reach any given goal state from a starting state. The main idea is learning to predict an intermediate state / midpoint between the current state and the goal, which can help with improving the learned quasi-metric value function. Unlike standard TD learning which updates the value function using neighbouring states, AML uses the predicted midpoints to propagate information over longer distances. The paper shows that across six different environments it consistently improves or matches a TD baseline. The improvements are more noticable in the larger maze environment.
Critical strengths
-
The paper identifies a limitation of TD learning for long-horizon tasks, and the main idea is well motivated.
-
AML improves over the quasi-metric TD baseline across the evaluated environments, especially on Ant Big Maze (RS). This shows that midpoint learning could be useful for tasks that might have a larger state space, or a long-horizon reasoning about the state space.
-
The proposed approach adds an auxiliary midpoint learning objective which seems easy to add to the general goal-reaching RL framework, and seems to be applicable to any other similar methods.
Critical weaknesses
-
The proposed idea has limited novelty in comparison to prior work, especially Kasaura et al. The main contribution of the paper seems to be adapting the approach of that paper (enforcing global consistency of a quasi-metric) to goal-reaching RL. While this is potentially valuable, I think the paper could benefit from a discussion of the technical challenges specific to the RL setting and which aspects of the proposed algorithm are necessary to address them.
-
There are a few important ablations missing from the paper. Specifically, 1) if we replace the midpoints by simpler states (e.g. random states along sampled trajectories), how do the results change? 2) Is the discriminator necessary? 3) How sensitive are the results to the weight of the additional objective?
-
The main motivation of the paper seems to be that TD learning can underestimate distances between states far from each other, and that midpoint learning can especially be useful for those settings. However, the experiments mainly focus on / evaluate policy success rather than the accuracy of the learned value function. I think this work could benefit from additional evals. For example, in a simple maze setup where the shortest distances can be computed, you could measure how similar the learned value function is to the ground-truth distance.
-
The paper is difficult to read, and lacks visualizations that can build intuition, e.g. visualizations of predicted midpoints.
Questions for authors
-
Why is minimizing the max of the two distances preferable to minimizing their difference or directly encouraging the two distances to be equal?
-
Have you tried any ablations suggested in weakness #2? e.g. replacing midpoints with simpler states, such as states sampled from previous trajectories?
-
What challenges do you expect when applying AML to more complex environments?
Update and Withdrawal
Thank all reviewers to your thoughtful and informative reviews!
To keep record it, we update our paper as we mentioned as our comment. We expanded related work sections to include missing prior research. We also added four Appendix:
However, we decided to withdraw our paper according to the results of additional experiment, shown in Appendix J. While our method outperformed RIS in Ant Big Maze (RS) environment, Online TRL, which samples intermediate points from past trajectories and uses them for update of V values instead of predicted midpoints, showed comparable performance to our method. These results raise questions about whether the effectiveness of the proposed method stems from the prediction of the midpoint. While these results themselves may be interesting and it remains possible that our method’s superiority will be demonstrated in a larger environment, such as OGBench giant maze suggested by Reviewer N2nH, we withdraw this paper because the paper's argument cannot be upheld as is. We aim to write a better paper based on further research.
We have learned a great many things about this field through reviews. We would like to express our gratitude to reviews once again.