acceptodds
Under review as a conference paper at ICLR 2027

Rich Outcomes, Selective Constraints: Goal Overspecification in Offline Goal-Conditioned Learning

60 views

Abstract

Reward-free offline goal-conditioned learning obtains generality by treating achieved outcomes as reusable supervision, enabling policies to learn from offline experience without task-specific rewards. Yet a complete achieved outcome can contain more information than a downstream task requires. We show that conflating what an outcome describes with what a task demands creates goal overspecification: full-state hindsight objectives can require reproducing properties that the current success criterion leaves unconstrained. This tension is particularly important for general-purpose learning, where such properties should be retained because they may define other tasks. We therefore separate outcome information from task constraints with Goal-Set Hindsight Relabeling (GS-HER). GS-HER pairs a reference outcome with a query that defines a task-dependent equivalence class and makes the goal-conditioning representation invariant to unqueried outcome properties, while preserving the base learner’s hindsight sampling, supervision, target construction, and optimization objective. Across five heterogeneous offline goal-conditioned learners and OGBench manipulation tasks, replacing full-state semantics with the oracle task projection yields higher mean success in all backbone–task pairs; GS-HER recovers much of this gap, increasing average success from to , compared with for the oracle projection. In contrast, gains largely disappear on PointMaze, where outcome and task semantics are already aligned. A single GS-HER checkpoint can additionally support multiple object- and robot-centric success predicates from the query family represented during training by changing only the inference-time query. Our results suggest a simple principle for general-purpose robot learning: retain outcome information broadly, but impose task constraints selectively.

Primary area:
reinforcement learning
Keywords:
offline reinforcement learning, reward-free learning, goal-conditioned reinforcement learning, hindsight relabeling, goal representations, goal specification

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qjxxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper tackles the goal specification problem in offline goal-conditioned RL. In offline GCRL, we typically train a GC policy with full goal states, but we often only care about a specific part of the state when evaluating its goal-reaching performance (e.g., the object position alone). The authors propose a method to specify such a goal state mask at test time. To do this, they train a goal mask-aware policy with random masks during training. At test time, they specify the mask given by the environment. The authors show that this test-time method often matches the oracle train-time method (HER-Task) across diverse OGBench tasks.

Critical strengths
  • I liked the idea of test-time masking. This makes the policy much more flexible than training a policy with a predefined goal mask/representation. The authors show that this test-time method roughly matches the performance of the more restrictive train-time equivalent.
  • The proposed method is simple and straightforward to implement.
Critical weaknesses
  • The paper still assumes access to the oracle mask at test time. It is unclear how such a mask can be provided for general or image-based tasks, where we do not have access to ground-truth information. It would have been much more convincing if the authors had shown that their method can work on pixel-based tasks too.
  • The proposed method is not fully unsupervised. During training, the authors seemingly use semi-handcrafted goal masks (e.g., contiguous blocks), which implicitly assume that state dimensions are structured in a certain manner. This assumption may not hold in general in practice (especially with visual observations).
  • (Minor?) Abstract and Intro seem AI-generated (as a reference, Pangram says they're 100% AI-generated). I know Pangram is not perfect and using AI to polish a paper might not necessarily be a problem by itself, but I just wanted to mention this observation anyway. Personally, I would have preferred an unedited, human-written draft to one that comes across as clearly AI-generated/polished.

In general, my main concern is about the applicability of the proposed method. I'd update the score if the authors show that their method can be applied to domains with fewer structural assumptions (e.g., pixel-based environments, state-based environments without contiguous assumptions, etc.).

Questions for authors
  • What exactly is "metadata" in Sec 3.3?
  • Have you evaluated GS-HER without the contiguous mask sampling part (70%), i.e., fully supervision-free masking during training? How does it degrade performance compared to GS-HER across the board?
3 comments

We thank reviewer qjxx for recognizing the flexibility and simplicity of GS-HER and for suggesting a concrete ablation. We address the questions below and provide preliminary results on the requested experiment.

Task specification at inference. We consider this a strength, not a weakness. We agree that GS-HER requires a user-specified query at inference. Every goal-conditioned controller requires a goal specification; our contribution is not automatic discovery or grounding of task-relevant factors. Rather, unlike HER-Task, which commits to a fixed oracle projection during training, GS-HER trains without downstream task labels and lets a single checkpoint support different task predicates at inference, without retraining. This is particularly relevant when downstream tasks are unknown during offline pretraining.

Structured queries versus supervision. We agree that contiguous-block sampling exploits an inductive bias: neighboring entries in a factored state vector may describe related physical properties. However, this is not task-specific supervision. Blockwise queries are generated procedurally, without rewards, task labels, or semantic factor annotations. Thus, GS-HER Blockwise is annotation-free, although not invariant to permutations of state-coordinate order. We acknowledge this assumption in App. D.1 and directly examine its importance below.

Random-only ablation. Following the reviewer's suggestion, we remove contiguous-block sampling entirely. This uses the random-coordinate distribution exclusively as suggested; it makes no assumptions about coordinate ordering.

Table R1. Official success (%), Random-only / Blockwise. Random-only is one training seed; Blockwise values are five-seed means from Table 2. A dash denotes a Random-only run not yet available. SP/SN = cube-single play/noisy; DP/DN = cube-double play/noisy; PM-M/PM-L = pointmaze medium/large; AM-M = antmaze medium.

Env.GCDLGCIVLGCIQLCRLGCBC
SP94.7/9692.3/9591.7/9269/7268.0/75
SN100/100100/10099.7/10086/8768.3/94
DP21.0/3657.0/6012.3/235.3/118.7/17
DN57.3/6653.7/5434.7/3812/64.0/10
Scene63.7/5071.3/8433.7/2812/617.3/12
PM-M90.3/9681.3/8953.3/5737.3/4325.7/22
PM-L80.7/7657.0/4524.7/2224.3/3218.0/25
AM-M93.3/9368.0/4438.3/2857.3/3464.0/45

Preliminary observations. (Table has been updated, will include final observations in the revised manuscript.) Without any coordinate-locality assumption, Random-only exceeds HER-Full in 21/22 completed manipulation comparisons. On these matched backbone-environment pairs, mean success is 52.5% (Random-only), 56.7% (Blockwise), versus 25.3% (HER-Full). Across the 13 completed navigation comparisons, Random-only averages 57.8% versus 52.% for Blockwise. These resuls suggest that GS-HER's gains do not depend exclusively on contiguous masks, although Blockwise remains stronger on average in manipulation. They are descriptive, not statistical comparisons: Random-only currently uses one seed, and some CRL evaluations are pending. We are completing five independent seeds and will replace the preliminary figures with means and standard deviations.

Visual domains. We agree that extending goal queries beyond factored states is important. A natural extension would apply queries to structured visual representations, potentially using spatial masking or learned object-centric factors. Mapping semantic task requirements to spatial coordinates introduces additional grounding challenges. This is indeed our current research direction, but beyond our present contribution. We hope to share relevant results in the near future. We deliberately use factor states to isolate goal specification without introducing additional representation-learning challenges.

Sec 3.3. Metadata. Learning specific metadata (ξ)(\xi) includes information such as the sampled future-state temporal offset. We will clarify the definition in the revision.

We acknowledge the feedback on writing and will streamline the Abstract and Introduction; AI-assisted language editing was disclosed and we find it a helpful tool as non-native English speakers.

We appreciate the reviewer's willingness to reconsider the rating in light of a coordinate-order-agnostic masking ablation.

Thanks for the response.

  • "Task specification at inference. We consider this a strength, not a weakness.": Why? My concern is that "it is unclear how such a mask can be provided for general or image-based tasks, where we do not have access to ground-truth information." How can it be interpreted as a strength? Note that I was not comparing to GS-HER in this comment. To me, it does require ground-truth state information and cannot directly be applied to pixel-based tasks, which is a substantial limitation of the proposed method.
  • "However, this is not task-specific supervision. Blockwise queries are generated procedurally, without rewards, task labels, or semantic factor annotations. Thus, GS-HER Blockwise is annotation-free, although not invariant to permutations of state-coordinate order." - I do think it is highly task-dependent supervision. It directly depends on how exactly state dimensions are ordered, which is typically done in a highly arbitrary manner. Some tasks may have a favorable dimension ordering, but other tasks may not.
  • I appreciate the new results with random-only ablation. It's nice to see that the proposed method doesn't entirely depend on blockwise maskings.

In general, I think I'm on the fence about this paper. On one hand, my main concern is partially addressed by the new result with random-only. On the other hand, the method has a clear methodological drawback that limits its applicability. I also agree with Reviewer HM8L that the manuscript is not very well-written and seemingly heavily AI-generated (in fact I felt the same for the rebuttal; Pangram agrees too). While I've updated my score to 3, I would have given it a 2.5 (borderline) if it were an option, and I wouldn't be strongly opposed to rejection either.

Thanks for the follow-up and reconsidering your initial assessment. We appreciate your continued engagement.

  1. Inference-time task specification and visual applicability. We did not mean to dismiss your concern in the first place. GS-HER currently assumes access to meaningful state variables and a way to specify which of them define the desired task. This is a genuine limitation, particularly for pixel-based observations where such factors are not directly available. We do not claim to have solved this grounding problem.

What we intended to emphasize is a narrower distinction: given an available task specification, GS-HER avoids committing the learning process to that specification during training. Unlike HER-Task, a single checkpoint can accomodate different goal constraints at inference. We believe this is a useful property within the studied setting, even though extending it to unstructured observations remains an open challenge worth pursuing.

We have revised Section 5 to make these even more explicit.

  1. Dependence on coordinate ordering.

We really appreciate your suggestion to evaluate on Random-only queries. It led to one of the most important additions to the revised manuscript.

  • GS-HER Random improves over HER-Full in all 25 manipulation comparisons.
  • Mean manipulation success increases from 24.4% to 52.%, compared with 56.5% for Blockwise.
  • Random training requires no contiguous-coordinate or semantic-grouping assumptions.

These results do not resolve the visual grounding limitation, but they establish that the observed improvement is not dependent on favorable coordinate ordering. We believe this substantially strengthens the central empirical finding.

  1. Presentation and contribution. We also took your comments on writing seriously. We have substantially rewritten the Introduction and technical exposition.

Ultimately, we agree that GS-HER is a simple intervention and that its current applicability is limited to factored state spaces. We do not dispute either point. However, we believe the contribution should also be evaluated for what it establishes within that scope: goal overspecification is a systematic performance bottleneck across heterogeneous offline GCRL methods, and much of this limitation can be mitigated without task-specific pretraining or assumptions about coordinate ordering. The ability to reuse a single checkpoint across different goal specifications provides an additional practical motivation.

We do not expect these revisions to eliminate every concern, particularly regarding future applicability to visual domains. Nevertheless, we hope the completed experiments, clearer methodological framing, and more explicit limitations provide sufficient evidence to support a favorable assessment of the contribution within its stated scope.

Thank you again for your constructive criticism. Your suggestions have materially improved the paper, and we genuinely appreciate the time you have invested in this discussion.

Reviewer_G2ZGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This work highlights goal overspecification as an important problem in goal-conditioned RL: relabeling schemes reuse whole states as goals, and policy are asked to reach configurations in every detail. For instance, a cube manipulation policy is asked to control not just the position of the cube, but that of the arm as well. The core algorithmic contribution is the introduction of a (partially learned) goal representation, which applies a binary mask over state and goal variables, and sums them up to learned masked embeddings. The mask can be sampled according to diverse schemes through training, and is compatible with most GCRL algorithms. The evaluation covers 8 tasks and 5 algorithmic backbones: masking the goal according to the task information always improve performance (demonstrating that goal over specification is indeed an issue), and the method induces similar improvements.

Critical strengths
  • This work is presented in a very clear and informative way.
  • The authors provided convincing evidence confirming the problem of goal over specification.
  • The experimental protocol is sound, and the evaluation comprehensive.
Critical weaknesses
  • The main weakness is that, to the best of my understanding, the method is essentially randomly sampling binary masks, and replacing the goals with their masked version, concatenated to the mask. As a result, the method is simply a conditional variant of HER-TASK. To clarify, let’s set ese_s and ege_g to 00 without loss of generality. Then the mapping between (s,ψGS(s,g,q))(s, \psi_{GS}(s,g,q)) and (s,q⋅g,q)(s, q \cdot g, q) is a fixed bijection. Therefore, providing the former as a goal representation is essentially the same as providing the latter, if I understand correctly. If this is correct, then the main contribution of this work are (i) highlighting goal-specification as an issue, which is somewhat well known and (ii) proposing a conditional version of the standard goal masking strategy that is used to mitigate this issue. If this is the case, I do not think this submission meets the originality criteria of the conference.
  • Independently from the previous point, the task-specific coordinates (e.g. cube position) still need to be known during inference. This means that the method requires as much prior knowledge as HER-TASK (only at inference time)
  • Following from the above weakness, extending this approach beyond easily interpretable state spaces, e.g. to images, does not seem trivial.

Minor:

  • Some sentences are hard to parse, for instance: “A single GS-HER checkpoint can additionally support multiple object- and robot-centric success predicates from the query family represented during training by changing only the inference-time query” is written before explaining what a query is. The same applies to the paragraph on line 105. Other words like “certifies” and “witnesses” are overly technical (line 189).
  • Table 2 would benefit from having average scores across environments.
  • The acknowledgement session should be removed altogether until acceptance.
  • Should the parameter counts on line 1280 be environment dependent?
  • Through the evaluation, I count 8 environments ant 5 algorithms, for a total of 40. I do not understand statements mentioning e.g. 24/25 combinations. Should it not be X/40?
Questions for authors
  • Can you elaborate on why GC-HER is not equivalent to a conditional variant of HER-TASK, as discussed above?
  • How would you apply GS-HER to visual domains?
9 comments

We thank the reviewer G2ZG for recognizing the clarity, experimental rigor, and convincing evidence of goal overspecification. We particularly appreciate the opportunity to clarify the intended contribution, as we believe the distinction between the simplicity of the implementation and the scientific findings deserves furthre emphasis.

Q1 - Equivalence, originality, and contribution. We agree with the reviewer's mathematical observation. For fixed inactive-coordinate embeddings, GS-HER's representation is equivalent to conditional masking with an explicit query. The learned embeddings improve optimization but do not alter the underlying goal semantics. We do not claim that binary masking itself is a novel operation.

However, we believe the contribution shoudl be understood beyond this individual operation. The central question investigated in our paper is whether the conventional practice of treating complete achieved outcomes as exact goals introduces a systematic limitation in offline goal-conditioned learning, and whether this limitation can be addressed without sacrificing task-agnostic pretraining.

Our results establish three complementary findings:

  1. Goal overspecification is a systematic empirical bottleneck across heterogeneous learning objectives.
  2. Task-selective goal semantics can be learned without committing to a downstream task during pretraining. Unlike HER-Task, which depends on a fixed oracle task projection, GS-HER trains across task-agnostic queries.
  3. This task-agnositc query sampling yields a single checkpoint that supports multiple downstream goal specifications without retraining.

We believe these findings provide a useful scientific contribution independently of the novelty of the masking operator: the paper identifies and systematically characterizes a limitation shared by diverse offline GCRL methods, formalizes its underlying goal-set semantics, and demonstrates a broadly compatible invervention that substantially mitigates it while preserving downstream flexibility.

Task specification at inference. We consider this a strength, not a weakness. We would like to distinguish specifying a task from committing the learning procedure to that task.

In goal-conditioned control, a desired objective must ultimately be provided. HER-Task uses this knowledge during training and consequently requires a separately trained model to change the task projection. GS-HER does not require this commitment: the same checkpoint can interpret different reference outcomes under different task constraints.

This distinction is particularly relevant to reusable offline pretraining, where the eventual downstream tasks may not be known when the data was collected or the model is trained. We do not claim automatic task discovery of task relevance; our contribution is to enable task-selective conditioning without task-specific retraining.

Q2 - Visual domains. A natural extension would apply queries to structured visual representations, potentially using spatial masking or learned object-centric factors. Mapping semantic task requirements to spatial coordinates introduces additional grounding challenges. This is indeed our current research direction, but beyond our present contribution. We hope to share relevant results in the near future. We deliberately use factor states to isolate goal specification without introducing additional representation-learning challenges.

Clarifications. We will introduce queries earlier and simplify technical terminology. We thank reviewer G2ZG for identifying this missing detail in the appendix: the reported parameter count correspond to cube-single-play and noisy environments, as they share the same state definition.

The 24/25 statistic refers explicitly to manipulation (5 environments x 5 backbones); The complete evaluation consists of 8 environments.

As requested, we provide the average success rates(%) calculated from Table 2 below.

\hline \text{Backbone / Scope} & \text{HER-Full} & \text{HER-Task} & \text{GS-Block} & \text{GS-Sem.} \\\\ \hline \text{GCDL (Manip.)} & 22.4 & 55.2 & 69.6 & 71.8 \\\\ \text{GCIVL (Manip.)} & 43.6 & 82.0 & 78.6 & 78.0 \\\\ \text{GCIQL (Manip.)} & 33.6 & 72.2 & 56.2 & 52.2 \\\\ \text{CRL (Manip.)} & 11.6 & 45.4 & 36.4 & 34.6 \\\\ \text{GCBC (Manip.)} & 11.0 & 45.4 & 41.6 & 37.8 \\\\ \hline \text{All Manip. (25)} & \mathbf{24.4} & \mathbf{60.0} & \mathbf{56.5} & \mathbf{54.9} \\\\ \text{All Nav. (15)} & \mathbf{50.6} & \mathbf{59.8} & \mathbf{50.1} & \mathbf{57.0} \\\\ \hline \end{array}$$ Finally, we appreciate the reviewer's assessment that the experiments are sound and the evidence convincing. Since the main concern appears to be originality, we would welcome clarification on whether the soundness rating reflects a separate methodologial concern. We hope this clarifications better convey our contribution and would appreciate your reconsideration.

Thank you for your clarification. Given that the method is a conditional variant of HER-Task, the complexity of the masking operation is Eq.3 is not entirely justified, i.e. why do additional embeddings need to be learned, when conditioning on a masked goal (and potentially on a binary mask) would suffice? I think that this additional complexity masks the simplicity of the approach proposed. I will keep my current score for the time being.

We thank the reviewer for the follow-up and appreciate the opportunity to clarify this point.

Regarding the learned embeddings. We agree that they are not mathematically necessary to represent the same goal sets. With an explicit binary query, replacing inactive coordinates with zero is informationally equivalent and a straightforward idea. However, there is an important distinction between representational equivalence and optimization behavior.

In continuous state spaces, zero can be a valid and meaningful feature value. Replacing all inactive coordinates by zero introduces an artificial concentration at that value, altering the distribution of the goal-conditioning inputs. Learnable embeddings instead allow the network to adapt these inactive-coordinate representations during training. Indeed, we also evaluated fixed-zero and fixed-random embeddings and found that learned embeddings performed better (App. G.1.). We have explicitly mentioned this desing choice in the updated manuscript Sec 3.2. Nevertheless, we emphasize that these embeddings are a minor implementation choice, not a claimed algorithmic novelty.

More importantly, we would like to emphasize that GS-HER's contribution extends beyond the masking operation itself. The essential distinction from HER-Task is not how inactive coordinates are represented, but how goal constraints are incorporated during training. While HER-Task commits to a fixed downstream task projection, GS-HER learns across task-agnostic queries, enabling a single checkpoint to support different downstream goal specifications without retraining.

We believe that reducing GS-HER to its masking operation does not fully capture the broader scientific contribution: systematically identifying goal overspecification as a limitation across heterogeneous offline GCRL methods, and demonstrating that task-selective goal conditioning can mitigate this limitation without sacrificing task-agnostic pretraining or downstream flexibility.

We hope this clarifications better convey our contribution and would appreciate your reconsideration. We also thank the reviewer for your time and constructive follow-up discussion, which has helped us improve the presentation of our contribution.

Reviewer_HM8LOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The idea of this paper is that, for goal-conditioned RL, it's important to manage the goal representation in a task-specific way. If the goals are fully-detailed states, then we are generally asking the learner to achieve irrelevant (or impossible) detail. If we "pare down" the goal representation (to name sets of states) in advance, then we avoid overspecification but we can't re-use our data. In this work, we save our data with fully detailed labels, but decide, based on the task, what factors (of a factored state representation) are relevant and use only those to specify the target goal set.

The contributions are this framing of the problem and an algorithm for carrying it out. The method is shown to be better than using fully detailed goals and sometimes quite close to the "oracle" that knows exactly what factors are relevant. This is not so surprising, in my view. But the last set of results, showing that this factored goal representation allows generalization to goal-factor specifications that were not encountered during training is interesting and shows a strength of the method.

Critical strengths

The ability to flexibly define goals at the target level of abstraction is important.

The fact that goal generalization arises (as shown in section 4.4) shows a strength of this method.

Critical weaknesses

The writing is both redundant and lacking in some important detail and set-up. It is hard to follow and reads kind of like Claude (which is fine---usage is disclosed by the authors---but nonetheless hard to read).

The test problems all seem fairly artificial. It's good to have some problems of this kind for debugging and intuition building, but it would be good to see method this demonstrating utility on a problem someone cares about.

I tried to read it several times, but the writing in the technical sections was just not clear enough for me to actually understand the detailed algorithmic contribution.

Minor:

  • Equation 2 seems to say roughly the same thing as equation 1 (but neither is set up well enough to be completely clear)
  • Remind us quickly of the basic operations of HER and what a "base relabeler" is
Questions for authors
  • Why does GS-HER ever outperform HeR-Task? Do we have any idea of the significance of these results?
  • What is the operator in equation 1? (should be defined!)
  • What is "learner-specific metadata"?
6 comments

We thank reviewer HM8L for the thoughtful feedback, particularly for recognizing the value of flexible goal specification and the results in Section 4.4. We appreciate the positive assessment of the paper's soundness.

Contribution. An important clarification regarding the "Paper Summary": GS-HER does not select task-relevant factors during training. Instead, it samples task-agnostic queries over complete hindsight outcomes. The downstream task is specified only at inference, allowing one checkpoint to support multiple goal specifications without retraining. This is the key distinction from fixed HER-Task projections.

Significance. While avoiding irrelevant goal constraints may seem intuitive once it is brought to light, its systematic impact across offline GCRL methods is an important empirical finding. HER-Task improves over full-state conditioning in 25/25 manipulation comparisons, and GS-HER improves in 24/25. Our deliberately simple intervention isolates this effect across five heterogeneous learners.

Experimental relevance. OGBench (ICLR 2025) is an established benchmark specifically designed to study offline GCRL challenges the community cares about, such as trajectory stitching, long-horizon reasoning, and manipulation. If by "a problem someone cares about" the reviewer means a real-world application, we agree that such validation would be valuable. However, we believe a controlled benchmark is better suited to isolating and investigating a general learning formulation.

Technical questions.

  • As noted in Section 4.2 (lines 311-312), HER-Task is an oracle projection, not a performance ceiling. Multi-query training may benefit from representation sharing, implicit curriculum... although these mechanisms have not been isolated.
  • The operator (⊙\odot) denotes the elementwise multiplication.
  • Learning specific metadata (ξ\xi) includes information such as the sampled future-state temporal offset. The base relabeler is simply the original learner's hindsight goal sampler.

We will revise the exposition to make it clearer. We will simplify Equations 1-2 as suggested, briefly introduce HER and the relabeling procedure for context, and streamline the technical sections.

We hope these clarifications better convey the contribution and would appreciate your reconsideration. Do not hesitate to follow-up in this thread for further clarifications/ recommendations.

AuthorsOpenReview ↗

Additional clarification: GS-HER through a concrete training example.

We would like to provide a concrete example of a single GS-HER training update, to help you understand the detailed algorithmic contribution.

Consider training GCIQL on an OGBench cube manipulation trajectory:

1. Standard hindsight relabeling (HER) (unchanged): The base learner samples a transition ((st,at,st+1))((s_t,a_t, s_{t+1})) and a future achieved state (g=st+20)(g=s_{t+20}). This reference outcome contains the complete state: robot configuration, end-effector pose, gripper state, cube pose, and velocities. The original relabeler also provides metadata ξ\xi such as the temporal offset (k=20)(k=20), which GCIQL uses in its native supervision.

2. Sample a task-agnostic query (GS-HER). Independently of the reference sampling, we sample a binary query (q∼pQ)(q \sim p_Q). For example, suppose (q)(q) activates only the three coordinates describing the cube position. The reference outcome is still the complete achieved state, but the query now defines the goal as reaching that cube position, regardless of the robot configuration, gripper state, or cube orientation.

3. Construct the goal-conditioning input. GS-HER replaces inactive coordinates in the goal-comparison representation with learned embeddings and supplies the query mask explicitly. Consequently, two reference outcomes with identical cube positions but different inactive properties produce identical goal-conditioning representations under this query. Importantly, the complete current state (s_t) remains available to the policy and value functions for control.

4. Apply the original learning update (unchanged). GCIQL receives the query-conditioned representation, but its objective, target reconstruction, and sampled temporal offset remain unchanged. The demonstrated trajectory still provides a valid reachability witness: the queried goal set was reached no later than (t+20). We do not introduce query-specific rewards, recompute first-hitting times or modify the underlying learning algorithm.

During training, this procedure is repeated with different sampled queries over the same offline trajectories. No downstream task labels or oracle task projections are used to select these queries.

At inference, the same trained checkpoint can be conditioned on cube position, end-effector pose, or other goal specifications within its represented query family simply by changing (q)(q), without retraining. This is the central distinction from HER-Task, which commits to one fixed projection before training, and HER-Full, which always conditions on the complete reference outcome.

In short, GS-HER changes neither which experience provides supervision nor how the base learner is optimized. It changes which properties of an achieved outcome are treated as requirements of the goal.

We hope this example makes the algorithmic contribution clearer to reviewer HM8L.

Reviewer_HM8LOpenReview ↗

That example is very helpful!

It would be a great addition to the paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised Manuscript – Summary of Changes and Thanks to Reviewers.

Dear Reviewers, We have uploaded a revised version of our manuscript incorporating the feedback and discusions from the review process.

First, we would like to sincerely thank all three reviewers for the time, care, and thoughtfulness invested in evaluating our work. Your comments have helped us to better present our contribution, clarify methodological details, and strengthen our experimental evidence.

We really appreciate the constructive and profesional engagement throughout this process. We feel fortunate to have received good feedback, and we believe the manuscript has improved considerably as a result.

For convenience, here is a brief list of main revisions:

  • Presentation and motivation (Abstract, Sections 1-2): Reworked exposition to explain goal overspecification more intuitively, distinguish our scientific contribution from the simplicity of the masking operation, and better position the work relative to existing literature. Also introduced HER briefly.
  • Methodological clarity (Section 3, Algorithm 1): Clarified the goal-set formulation, query-conditioned invariance, original HER supervision, learner-specific metadata, and the role of learned embeddings. We also explicitly distinguish representational equivalence from optimization considerations.We have also introduced Algorithm 1 to better comunicate GS-HER minimal intervention.
  • New Random only experiments (Section 4, Table 2): Completed the requested coordinate-order-agnostic ablation across all five learners and eight environments, with five independent seeds. Without contiguous-block or semantic priors, GS-HER Random improves over HER-Full in all 25 manipulation comparisons.
  • Experimental reporting (Figures 2-3, Table 2): Updated the complete results and aggregate comparisons, added mean performance across environments, and clarified the distinction between the 25 manipulation and 40 total learner–environment comparisons.
  • Scope and limitations (Section 5): More explicitly acknowledged the reliance on factored state representations, the need for task specification at inference, and the challenges of extending the approach to visual and other unstructured domains.

Beyond factored states: scope and applicability. The reviewers acknowledge the importance of goal overspecification , while raising concerns about its applicability beyond factored state spaces, particularly to image-based observations. We agree that extending GS-HER benefits beyond factored states is an important challenge. A practical extension is to use perception models to extract structured, object-centric representations from images, enabling GS-HER to operate within perception-based robotic systems, including those using agentic foundation models for planning and control. While this approach depends on the quality of the extracted features, it provides a route toward real-world applications. Learning queryable goal representations directly from pixels remains an important direction for future work.

Once again, thank you for your constructive criticism and thoughtful engagement. We truly appreciate the opportunity to improve our work through this discussion and welcome any further comments or suggestions.

The authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.