- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper tackles the goal specification problem in offline goal-conditioned RL. In offline GCRL, we typically train a GC policy with full goal states, but we often only care about a specific part of the state when evaluating its goal-reaching performance (e.g., the object position alone). The authors propose a method to specify such a goal state mask at test time. To do this, they train a goal mask-aware policy with random masks during training. At test time, they specify the mask given by the environment. The authors show that this test-time method often matches the oracle train-time method (HER-Task) across diverse OGBench tasks.
Critical strengths
- I liked the idea of test-time masking. This makes the policy much more flexible than training a policy with a predefined goal mask/representation. The authors show that this test-time method roughly matches the performance of the more restrictive train-time equivalent.
- The proposed method is simple and straightforward to implement.
Critical weaknesses
- The paper still assumes access to the oracle mask at test time. It is unclear how such a mask can be provided for general or image-based tasks, where we do not have access to ground-truth information. It would have been much more convincing if the authors had shown that their method can work on pixel-based tasks too.
- The proposed method is not fully unsupervised. During training, the authors seemingly use semi-handcrafted goal masks (e.g., contiguous blocks), which implicitly assume that state dimensions are structured in a certain manner. This assumption may not hold in general in practice (especially with visual observations).
- (Minor?) Abstract and Intro seem AI-generated (as a reference, Pangram says they're 100% AI-generated). I know Pangram is not perfect and using AI to polish a paper might not necessarily be a problem by itself, but I just wanted to mention this observation anyway. Personally, I would have preferred an unedited, human-written draft to one that comes across as clearly AI-generated/polished.
In general, my main concern is about the applicability of the proposed method. I'd update the score if the authors show that their method can be applied to domains with fewer structural assumptions (e.g., pixel-based environments, state-based environments without contiguous assumptions, etc.).
Questions for authors
- What exactly is "metadata" in Sec 3.3?
- Have you evaluated GS-HER without the contiguous mask sampling part (70%), i.e., fully supervision-free masking during training? How does it degrade performance compared to GS-HER across the board?
3 comments
Thanks for the response.
- "Task specification at inference. We consider this a strength, not a weakness.": Why? My concern is that "it is unclear how such a mask can be provided for general or image-based tasks, where we do not have access to ground-truth information." How can it be interpreted as a strength? Note that I was not comparing to GS-HER in this comment. To me, it does require ground-truth state information and cannot directly be applied to pixel-based tasks, which is a substantial limitation of the proposed method.
- "However, this is not task-specific supervision. Blockwise queries are generated procedurally, without rewards, task labels, or semantic factor annotations. Thus, GS-HER Blockwise is annotation-free, although not invariant to permutations of state-coordinate order." - I do think it is highly task-dependent supervision. It directly depends on how exactly state dimensions are ordered, which is typically done in a highly arbitrary manner. Some tasks may have a favorable dimension ordering, but other tasks may not.
- I appreciate the new results with random-only ablation. It's nice to see that the proposed method doesn't entirely depend on blockwise maskings.
In general, I think I'm on the fence about this paper. On one hand, my main concern is partially addressed by the new result with random-only. On the other hand, the method has a clear methodological drawback that limits its applicability. I also agree with Reviewer HM8L that the manuscript is not very well-written and seemingly heavily AI-generated (in fact I felt the same for the rebuttal; Pangram agrees too). While I've updated my score to 3, I would have given it a 2.5 (borderline) if it were an option, and I wouldn't be strongly opposed to rejection either.
Thanks for the follow-up and reconsidering your initial assessment. We appreciate your continued engagement.
- Inference-time task specification and visual applicability. We did not mean to dismiss your concern in the first place. GS-HER currently assumes access to meaningful state variables and a way to specify which of them define the desired task. This is a genuine limitation, particularly for pixel-based observations where such factors are not directly available. We do not claim to have solved this grounding problem.
What we intended to emphasize is a narrower distinction: given an available task specification, GS-HER avoids committing the learning process to that specification during training. Unlike HER-Task, a single checkpoint can accomodate different goal constraints at inference. We believe this is a useful property within the studied setting, even though extending it to unstructured observations remains an open challenge worth pursuing.
We have revised Section 5 to make these even more explicit.
- Dependence on coordinate ordering.
We really appreciate your suggestion to evaluate on Random-only queries. It led to one of the most important additions to the revised manuscript.
- GS-HER Random improves over HER-Full in all 25 manipulation comparisons.
- Mean manipulation success increases from 24.4% to 52.%, compared with 56.5% for Blockwise.
- Random training requires no contiguous-coordinate or semantic-grouping assumptions.
These results do not resolve the visual grounding limitation, but they establish that the observed improvement is not dependent on favorable coordinate ordering. We believe this substantially strengthens the central empirical finding.
- Presentation and contribution. We also took your comments on writing seriously. We have substantially rewritten the Introduction and technical exposition.
Ultimately, we agree that GS-HER is a simple intervention and that its current applicability is limited to factored state spaces. We do not dispute either point. However, we believe the contribution should also be evaluated for what it establishes within that scope: goal overspecification is a systematic performance bottleneck across heterogeneous offline GCRL methods, and much of this limitation can be mitigated without task-specific pretraining or assumptions about coordinate ordering. The ability to reuse a single checkpoint across different goal specifications provides an additional practical motivation.
We do not expect these revisions to eliminate every concern, particularly regarding future applicability to visual domains. Nevertheless, we hope the completed experiments, clearer methodological framing, and more explicit limitations provide sufficient evidence to support a favorable assessment of the contribution within its stated scope.
Thank you again for your constructive criticism. Your suggestions have materially improved the paper, and we genuinely appreciate the time you have invested in this discussion.
We thank reviewer qjxx for recognizing the flexibility and simplicity of GS-HER and for suggesting a concrete ablation. We address the questions below and provide preliminary results on the requested experiment.
Task specification at inference. We consider this a strength, not a weakness. We agree that GS-HER requires a user-specified query at inference. Every goal-conditioned controller requires a goal specification; our contribution is not automatic discovery or grounding of task-relevant factors. Rather, unlike HER-Task, which commits to a fixed oracle projection during training, GS-HER trains without downstream task labels and lets a single checkpoint support different task predicates at inference, without retraining. This is particularly relevant when downstream tasks are unknown during offline pretraining.
Structured queries versus supervision. We agree that contiguous-block sampling exploits an inductive bias: neighboring entries in a factored state vector may describe related physical properties. However, this is not task-specific supervision. Blockwise queries are generated procedurally, without rewards, task labels, or semantic factor annotations. Thus, GS-HER Blockwise is annotation-free, although not invariant to permutations of state-coordinate order. We acknowledge this assumption in App. D.1 and directly examine its importance below.
Random-only ablation. Following the reviewer's suggestion, we remove contiguous-block sampling entirely. This uses the random-coordinate distribution exclusively as suggested; it makes no assumptions about coordinate ordering.
Table R1. Official success (%), Random-only / Blockwise. Random-only is one training seed; Blockwise values are five-seed means from Table 2. A dash denotes a Random-only run not yet available. SP/SN = cube-single play/noisy; DP/DN = cube-double play/noisy; PM-M/PM-L = pointmaze medium/large; AM-M = antmaze medium.
Preliminary observations. (Table has been updated, will include final observations in the revised manuscript.) Without any coordinate-locality assumption, Random-only exceeds HER-Full in 21/22 completed manipulation comparisons. On these matched backbone-environment pairs, mean success is 52.5% (Random-only), 56.7% (Blockwise), versus 25.3% (HER-Full). Across the 13 completed navigation comparisons, Random-only averages 57.8% versus 52.% for Blockwise. These resuls suggest that GS-HER's gains do not depend exclusively on contiguous masks, although Blockwise remains stronger on average in manipulation. They are descriptive, not statistical comparisons: Random-only currently uses one seed, and some CRL evaluations are pending. We are completing five independent seeds and will replace the preliminary figures with means and standard deviations.
Visual domains. We agree that extending goal queries beyond factored states is important. A natural extension would apply queries to structured visual representations, potentially using spatial masking or learned object-centric factors. Mapping semantic task requirements to spatial coordinates introduces additional grounding challenges. This is indeed our current research direction, but beyond our present contribution. We hope to share relevant results in the near future. We deliberately use factor states to isolate goal specification without introducing additional representation-learning challenges.
Sec 3.3. Metadata. Learning specific metadata (ξ) includes information such as the sampled future-state temporal offset. We will clarify the definition in the revision.
We acknowledge the feedback on writing and will streamline the Abstract and Introduction; AI-assisted language editing was disclosed and we find it a helpful tool as non-native English speakers.
We appreciate the reviewer's willingness to reconsider the rating in light of a coordinate-order-agnostic masking ablation.