- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper adress a missmatch between the training and testing distribution of goal conditioned reinforcemente learning algorithms trained from state action trajectories. These algorithms usually learn via a self supervised learning objective that assumes that states closed in time are positive pairs at short distance and push any other state, taking any state as a possible goal. This creates a missmatch since at inference usually the goal is defined as a set of possible goals for example all the states sharing the same (x, y) position but do not care about the velocity at which you reach the goal. The paper assume to have knowledge about this features split and proposed a fix that adress this problem by selecting the closest goal and optimize to that instead of considering a random goal velocity to fill the observation gap.
Critical strengths
-
The paper is sound and adresses one of the know limitation of training with observed state as goals.
-
The method is simple and showed good empirical results in an extensive set of evaluation environments.
Critical weaknesses
-
The method is trivial and potentially a well known solutions, nonetheless I'm not aware of any published work about this.
-
The method assumes to have access to the information about task-provided splits that is a strong assumption that limit the application of the approach to factored observation, and make impossible to apply the method to the interesting case of high-dimensional observations. I think a much more interesting setting for this problem would be to learn from a limited dataset of (goal, goal_desired) paired observations how to regress the goal_desired to the closest goal.
Questions for authors
-
I think the related work is missing some other references learning the temporal distance. Like:
-
HIQL: Offline Goal-Conditioned RL with Latent States as Actions
-
Foundation Policies with Hilbert Representations
-
State Representation Learning for Goal-Conditioned Reinforcement Learning
-
Learning The Minimum Action Distance
2 comments
Thank you for the response. My main concern about the paper is the strong assumption about task-provided splits that limit the application of the approach, I will mantain my score for the moment and looking forward to discuss with other reviewers and AC discussion whether the paper make it to the quality bar for a top-tier conference.
Response to Reviewer TuLc: Originality, Applicability, and Related Work
Thank you for carefully reviewing our paper and for your helpful suggestions.
Response to the “Questions for Authors” section.
Thank you for pointing out these omissions in our related work. We have added the missing references and expanded the discussion of distance representations and goal-conditioned value learning in the revised manuscript. However, we are not yet clear on the specific question you would like us to address in this section. Could you please clarify it so that we can respond directly?
Response to Critical Weakness 1.
We agree that the method is simple. Simplicity alone, however, does not imply a lack of originality. Although using only specified coordinates as goal inputs is an established practice, we are not aware of prior work combining split state encoding with training-time latent completion search. Specifically, SSE in AQRL separately encodes specified and free coordinates, retaining both parts for state-to-state cost estimation while conditioning the actor's goal input only on the specified part. ANC then fixes the specified-coordinate latent and optimizes the free-coordinate latent to seek a completion with lower estimated reaching cost from the predicted next state for actor training.
Response to Critical Weakness 2.
We acknowledge that requiring a task-provided split limits the applicability of our method. Our experiments focus on vector-state observations with known splits. AQRL is difficult to apply directly to tasks where specified and free coordinates cannot be clearly distinguished, as is often the case with image observations. However, we would like to clarify that the formulation does not require low-dimensional states: it also applies to high-dimensional state vectors when the split between specified and free coordinates is known.
Regarding your suggested paired-observation setting, is your aim to address this applicability limitation? Specifically, do you mean learning a model from paired observations that directly maps a desired task goal to a closer full goal state or goal representation that satisfies the same task requirements? Would the model take specified-coordinate values as input, or directly use images or full state observations without a predefined split?