- Rating
- 2
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the problem of learning skills that can be reused in environments with different structures after being discovered in one environment. To this end, inspired by the bisimulation literature, it learns state representations that reflect action similarity and trains a network that captures temporal distance based on these representations. It then learns a policy based on HILP [1] that takes latent states as input to reduce dependence on specific maze layouts. Experiments evaluate the method through high-level policy learning and test-time planning in maze environments, demonstrating improved transfer performance to new mazes.
[1] Park, Seohong, et al. “Foundation Policies with Hilbert Representations.” 2024.
Critical strengths
- The motivation is clear: existing offline RL and offline skill discovery methods, such as HILP, struggle to transfer learned skills.
- The paper is easy to read, and the proposed method is well-designed to address the problem it aims to solve.
Critical weaknesses
- Although described in the limitations, the lack of manipulation experiments raises questions about the method’s scalability. In maze environments, the task-specific state typically consists of only the two-dimensional “xy-coordinates” within the full state space (e.g., 29 dimensions for Ant), making the network relatively easy to learn. In contrast, manipulation requires jointly considering object positions, the end-effector’s position and orientation, and other factors, making the network harder to learn. Although the algorithm does not rely on maze-specific assumptions, its performance may be due to the proposed loss being particularly easy to optimize in maze environments.
- The experimental setup for layout generalization in Section 5.3 does not seem fair. HILP uses the planning method presented in its original paper [1], whereas the proposed algorithm uses a graph-based method. This makes it difficult to fully distinguish whether the improvement in test-time performance comes from planning or from the learned representations. Since both methods learn state embeddings that can be used for graph-based planning, it seems reasonable to compare them using the same planning method.
- In Equation 3, using the L1 distance between actions to learn action-aware representations seems to have limited scalability. For example, when a robot is controlled through torques, the resulting motion depends heavily on the dynamics and its current pose. Thus, actions that are close in L1 distance may produce very different motions when applied in different states. In maze environments, this may not be a major issue because a single action has only a small effect on the task-relevant state, namely the xy-coordinates. However, it could make a substantial difference in manipulation, where precise contact is important.
For these reasons, I currently lean toward rejection.
[1] Park, Seohong, et al. “Foundation Policies with Hilbert Representations.” 2024.
Questions for authors
Most of my questions stem from the weaknesses discussed above. The main questions are as follows:
- Does the performance advantage in Section 5.3 remain when HILP and the proposed method use the same graph-based planner?
- What evidence supports the method’s applicability beyond mazes, particularly to manipulation, where task-relevant states are more complex and the L1 distance between actions may not reflect their effects due to differences in dynamics and the current pose?
- In Equation 6, is sampled uniformly from the action space, whereas Algorithm 1 samples it from the dataset. Could you clarify which sampling distribution is used? If actions are sampled uniformly from the action space, this may frequently produce actions outside the dataset distribution and lead to unstable training. Have you considered alternatives, such as sampling from a policy network?
3 comments
Response to Authors
We appreciate your time and effort in conducting the additional experiments. Your new results have addressed most of my concerns.
However, regarding manipulation tasks (see Weaknesses 1 and 3), although USB+ATR achieves the highest average performance, the improvement over the baselines is not substantial, and the experiments are limited to single-cube manipulation tasks. Therefore, I think that we should be somewhat cautious about claiming that the method generally performs well on manipulation tasks.
I understand that the rebuttal period is limited, so I do not expect additional experiments at this stage. However, I believe that demonstrating generalization to multi-block manipulation tasks would further strengthen the paper.
Overall, I appreciate your efforts in addressing my concerns. At this stage, I will maintain my original score.
Response to Reviewer Pwn5
Dear Reviewer Pwn5,
Thank you for your thoughtful feedback and for acknowledging our efforts to address your concerns. We appreciate your constructive suggestions, which have helped us further strengthen our evaluation.
We agree that evaluating our method on more challenging manipulation tasks would provide stronger evidence of its generalization capabilities. Motivated by your suggestion, we conducted additional preliminary experiments on the OGBench cube-double environment, which involves manipulating multiple objects and requires more complex interactions than the single-cube setting.
In this environment, USB+ATR achieves a preliminary success rate of 79.2 ± 4.2% with high-level policy training (2 random seeds), compared to 100.0 ± 0.0% on the corresponding cube-single-noisy environment. These results are encouraging and suggest that our method retains its ability to learn useful skills in more complex manipulation settings. Furthermore, we observed interesting learned behaviors involving interactions between the two cubes, providing additional qualitative evidence of the method's ability to discover meaningful manipulation skills.
We acknowledge that these results are preliminary, and we are conducting further evaluations with additional random seeds and more comprehensive baseline comparisons. We intend to include these results and analyses in the final version of the manuscript.
We sincerely appreciate your time, constructive feedback, and careful consideration of our work.
Response to Reviewer Pwn5
We thank the reviewer for the valuable feedback.
Q1: We agree with the reviewer that using different planning methods may introduce a potential source of unfairness in the comparison. Therefore, following the reviewer's suggestion to evaluate HILP with graph-based planning, we report the layout generalization results in the table below. The results show that incorporating graph-based planning improves HILP's performance. Although HILP achieves more promising results on the highly noisy explore dataset, consistent with the findings from the hierarchical evaluation in Table 1, USB+ATR still significantly outperforms HILP on average.
Q2: To further address the generalizability of our method beyond locomotion, we have conducted additional experiments and reported the results in a separate comment. These experiments further demonstrate the applicability of our method to other domains, particularly manipulation tasks that involve more complex skills.
Q3: Thank you for your detailed review. As you mentioned, to avoid sampling out-of-distribution actions, we sample actions from the dataset in practice.
Following your suggestion, we further ablate the choice of action distribution. Specifically, in this new experiment, when calculating d(si;sj) in Equation 6, we sample only actions that have been taken at either si or sj in the dataset. This experiment is closely related to the alternative you mentioned, namely, sampling actions from a policy network, as these actions are sampled from the goal-conditioned policy, starting from either si or sj and conditioned on arbitrary goals.
We evaluate this variant using test-time planning in the antmaze-large-navigate environment and report the results averaged over 5 random seeds: